[{"id":1763561,"web_url":"http://patchwork.ozlabs.org/comment/1763561/","msgid":"<2ed7620d-87be-74cd-f346-70f08e31a2d0@linaro.org>","list_archive_url":null,"date":"2017-09-05T18:40:27","subject":"Re: [PATCH] nss_files: Avoid large buffers with many host addresses\n\t[BZ #22078]","submitter":{"id":66065,"url":"http://patchwork.ozlabs.org/api/people/66065/","name":"Adhemerval Zanella Netto","email":"adhemerval.zanella@linaro.org"},"content":"On 04/09/2017 14:32, Florian Weimer wrote:\n> The previous implementation had at least a quadratic space\n> requirement in the number of host addresses and aliases.\n> \n> 2017-09-04 Florian Weimer \n> \n> \t[BZ #22078]\n> \tAvoid large NSS buffers with many addresses, aliases.\n> \t* nss/nss_files/files-hosts.c (gethostbyname3_multi): Rewrite\n> \tusing dynarrays and struct alloc_buffer.\n> \t* nss/Makefile (tests): Add tst-nss-files-hosts-multi.\n> \t(tst-nss-files-hosts-multi): Link with -ldl.\n> \t* nss/tst-nss-files-hosts-multi.c: New file.\n> \n> diff --git a/nss/Makefile b/nss/Makefile\n> index c9a5200f96..ecf3530188 100644\n> --- a/nss/Makefile\n> +++ b/nss/Makefile\n> @@ -63,6 +63,7 @@ xtests\t\t\t= bug-erange\n> # Tests which need libdl\n> ifeq (yes,$(build-shared))\n> tests += tst-nss-files-hosts-erange\n> +tests += tst-nss-files-hosts-multi\n> endif\n> \n> # If we have a thread library then we can test cancellation against\n> @@ -163,3 +164,4 @@ $(objpfx)tst-cancel-getpwuid_r: $(shared-thread-library)\n> endif\n> \n> $(objpfx)tst-nss-files-hosts-erange: $(libdl)\n> +$(objpfx)tst-nss-files-hosts-multi: $(libdl)\n> diff --git a/nss/nss_files/files-hosts.c b/nss/nss_files/files-hosts.c\n> index c2cd07584c..f40673056b 100644\n> --- a/nss/nss_files/files-hosts.c\n> +++ b/nss/nss_files/files-hosts.c\n> @@ -23,6 +23,7 @@\n> #include \n> #include \n> #include \n> +#include \n> \n> \n> /* Get implementation for some internal functions. */\n> @@ -116,24 +117,46 @@ DB_LOOKUP (hostbyaddr, ,,,\n> \t }, const void *addr, socklen_t len, int af)\n> #undef EXTRA_ARGS_VALUE\n> \n> +/* Type of the address and alias arrays. */\n> +#define DYNARRAY_STRUCT array\n> +#define DYNARRAY_ELEMENT char *\n> +#define DYNARRAY_PREFIX array_\n> +#include \n> +\n\nThis will create a 16 elements vector as default (128 bytes). Should we\naim to use the default or can we use a slight large buffer to speed up\nslightly generic case (assuming generic case use more than 16 entries)?\n\n> static enum nss_status\n> gethostbyname3_multi (FILE * stream, const char *name, int af,\n> \t\t struct hostent *result, char *buffer, size_t buflen,\n> \t\t int *errnop, int *herrnop, int flags)\n> {\n> + assert (af == AF_INET || af == AF_INET6);\n> +\n> /* We have to get all host entries from the file. */\n> struct scratch_buffer tmp_buffer;\n> scratch_buffer_init (&tmp_buffer);\n> struct hostent tmp_result_buf;\n> - int naddrs = 1;\n> - int naliases = 0;\n> - char *bufferend;\n> + struct array addresses;\n> + array_init (&addresses);\n> + struct array aliases;\n> + array_init (&aliases);\n> enum nss_status status;\n> + bool new_data = false;\n> \n> - while (result->h_aliases[naliases] != NULL)\n> - ++naliases;\n> + /* The output buffer uses the unused space after the first\n> + result. */\n> + struct alloc_buffer outbuf;\n\nIndentation seems off here.\n\n> + {\n> + /* Preserve the aliases encountered so far. */\n> + size_t i;\n> + for (i = 0; result->h_aliases[i] != NULL; ++i)\n> + array_add (&aliases, result->h_aliases[i]);\n> \n> - bufferend = (char *) &result->h_aliases[naliases + 1];\n> + char *bufferend = (char *) &result->h_aliases[i + 1];\n> + outbuf = alloc_buffer_create (bufferend, buffer + buflen - bufferend);\n> + }\n> +\n> + /* Preserve the addresses. */\n> + for (size_t i = 0; result->h_addr_list[i] != NULL; ++i)\n> + array_add (&addresses, result->h_addr_list[i]);\n> \n> again:\n> while ((status = internal_getent (stream, &tmp_result_buf, tmp_buffer.data,\n> @@ -154,110 +177,72 @@ gethostbyname3_multi (FILE * stream, const char *name, int af,\n> \t}\n> while ((matches = 0));\n> \n> + /* If the line matches, we need to copy the addresses and\n> +\t aliases, so that we can reuse tmp_buffer for the next\n> +\t line. */\n> if (matches)\n> \t{\n> -\t /* We could be very clever and try to recycle a few bytes\n> -\t in the buffer instead of generating new arrays. But\n> -\t we are not doing this here since it's more work than\n> -\t it's worth. Simply let the user provide a bit bigger\n> -\t buffer. */\n> -\t char **new_h_addr_list;\n> -\t char **new_h_aliases;\n> -\t int newaliases = 0;\n> -\t size_t newstrlen = 0;\n> -\t int cnt;\n> +\t new_data = true;\n> \n> -\t /* Count the new aliases and the length of the strings. */\n> -\t while (tmp_result_buf.h_aliases[newaliases] != NULL)\n> +\t /* Record the addresses. */\n> +\t for (int i = 0; tmp_result_buf.h_addr_list[i] != NULL; ++i)\n> \t {\n> -\t char *cp = tmp_result_buf.h_aliases[newaliases];\n> -\t ++newaliases;\n> -\t newstrlen += strlen (cp) + 1;\n> +\t /* Allocate the target space in the output buffer,\n> +\t\t depending on the address family. */\n> +\t void *target;\n> +\t if (af == AF_INET)\n> +\t\t{\n> +\t\t assert (tmp_result_buf.h_length == 4);\n> +\t\t target = alloc_buffer_alloc (&outbuf, struct in_addr);\n> +\t\t}\n> +\t else if (af == AF_INET6)\n> +\t\t{\n> +\t\t assert (tmp_result_buf.h_length == 16);\n> +\t\t target = alloc_buffer_alloc (&outbuf, struct in6_addr);\n> +\t\t}\n> +\t else\n> +\t\t__builtin_unreachable ();\n> +\n> +\t if (target == NULL)\n> +\t\t{\n> +\t\t /* Request a larger output buffer. */\n> +\t\t *errnop = ERANGE;\n> +\t\t *herrnop = NETDB_INTERNAL;\n> +\t\t status = NSS_STATUS_TRYAGAIN;\n> +\t\t goto out;\n> +\t\t}\n> +\t memcpy (target, tmp_result_buf.h_addr_list[i],\n> +\t\t tmp_result_buf.h_length);\n> +\t array_add (&addresses, target);\n> \t }\n> -\t /* If the real name is different add it also to the\n> -\t aliases. This means that there is a duplication\n> -\t in the alias list but this is really the user's\n> -\t problem. */\n> -\t if (strcmp (old_result->h_name,\n> -\t\t tmp_result_buf.h_name) != 0)\n> +\n> +\t /* Record the aliases. */\n> +\t for (int i = 0; tmp_result_buf.h_aliases[i] != NULL; ++i)\n\nShouldn't it use size_t for index?\n\n> \t {\n> -\t ++newaliases;\n> -\t newstrlen += strlen (tmp_result_buf.h_name) + 1;\n> +\t char *alias = tmp_result_buf.h_aliases[i];\n> +\t array_add (&aliases, alloc_buffer_copy_string (&outbuf, alias));\n\nShouldn't it check and bail for failed allocation for alloc_buffer_copy_string ?\n\n> \t }\n> \n> -\t /* Make sure bufferend is aligned. */\n> -\t assert ((bufferend - (char *) 0) % sizeof (char *) == 0);\n> +\t /* If the real name is different add it also to the aliases.\n> +\t This means that there is a duplication in the alias list\n> +\t but this is really the user's problem. */\n> +\t {\n> +\t char *new_name = tmp_result_buf.h_name;\n> +\t if (strcmp (old_result->h_name, new_name) != 0)\n> +\t array_add (&aliases,\n> +\t\t\t alloc_buffer_copy_string (&outbuf, new_name));\n> +\t }\n\nSame as before.\n\n> \n> -\t /* Now we can check whether the buffer is large enough.\n> -\t 16 is the maximal size of the IP address. */\n> -\t if (bufferend + 16 + (naddrs + 2) * sizeof (char *)\n> -\t + roundup (newstrlen, sizeof (char *))\n> -\t + (naliases + newaliases + 1) * sizeof (char *)\n> -\t >= buffer + buflen)\n> +\t /* Memory allocation failures during the expansion of the\n> +\t temporary arrays. */\n> +\t if (array_has_failed (&addresses) || array_has_failed (&aliases))\n> \t {\n> -\t *errnop = ERANGE;\n> +\t *errnop = errno;\n> \t *herrnop = NETDB_INTERNAL;\n> -\t status = NSS_STATUS_TRYAGAIN;\n> +\t status = NSS_STATUS_UNAVAIL;\n> \t goto out;\n> \t }\n> \n> -\t new_h_addr_list =\n> -\t (char **) (bufferend\n> -\t\t + roundup (newstrlen, sizeof (char *))\n> -\t\t + 16);\n> -\t new_h_aliases =\n> -\t (char **) ((char *) new_h_addr_list\n> -\t\t + (naddrs + 2) * sizeof (char *));\n> -\n> -\t /* Copy the old data in the new arrays. */\n> -\t for (cnt = 0; cnt < naddrs; ++cnt)\n> -\t new_h_addr_list[cnt] = old_result->h_addr_list[cnt];\n> -\n> -\t for (cnt = 0; cnt < naliases; ++cnt)\n> -\t new_h_aliases[cnt] = old_result->h_aliases[cnt];\n> -\n> -\t /* Store the new strings. */\n> -\t cnt = 0;\n> -\t while (tmp_result_buf.h_aliases[cnt] != NULL)\n> -\t {\n> -\t new_h_aliases[naliases++] = bufferend;\n> -\t bufferend = (__stpcpy (bufferend,\n> -\t\t\t\t tmp_result_buf.h_aliases[cnt])\n> -\t\t\t + 1);\n> -\t ++cnt;\n> -\t }\n> -\n> -\t if (cnt < newaliases)\n> -\t {\n> -\t new_h_aliases[naliases++] = bufferend;\n> -\t bufferend = __stpcpy (bufferend,\n> -\t\t\t\t tmp_result_buf.h_name) + 1;\n> -\t }\n> -\n> -\t /* Final NULL pointer. */\n> -\t new_h_aliases[naliases] = NULL;\n> -\n> -\t /* Round up the buffer end address. */\n> -\t bufferend += (sizeof (char *)\n> -\t\t\t- ((bufferend - (char *) 0)\n> -\t\t\t % sizeof (char *))) % sizeof (char *);\n> -\n> -\t /* Now the new address. */\n> -\t new_h_addr_list[naddrs++] =\n> -\t memcpy (bufferend, tmp_result_buf.h_addr,\n> -\t\t tmp_result_buf.h_length);\n> -\n> -\t /* Also here a final NULL pointer. */\n> -\t new_h_addr_list[naddrs] = NULL;\n> -\n> -\t /* Store the new array pointers. */\n> -\t old_result->h_aliases = new_h_aliases;\n> -\t old_result->h_addr_list = new_h_addr_list;\n> -\n> -\t /* Compute the new buffer end. */\n> -\t bufferend = (char *) &new_h_aliases[naliases + 1];\n> -\t assert (bufferend <= buffer + buflen);\n> -\n> \t result = old_result;\n> \t}\n> }\n> @@ -266,16 +251,52 @@ gethostbyname3_multi (FILE * stream, const char *name, int af,\n> {\n> if (!scratch_buffer_grow (&tmp_buffer))\n> \t{\n> +\t *errnop = errno;\n> \t *herrnop = NETDB_INTERNAL;\n> -\t status = NSS_STATUS_TRYAGAIN;\n> +\t status = NSS_STATUS_UNAVAIL;\n> \t}\n> else\n> \tgoto again;\n> }\n> else\n> - status = NSS_STATUS_SUCCESS;\n> + {\n> + /* Copy the address and alias arrays into the output buffer and\n> +\t add NULL terminators. The pointed-to elements were directly\n> +\t written into the output buffer above and do not need to be\n> +\t copied again. */\n> + if (new_data)\n> +\t{\n> +\t size_t addresses_count = array_size (&addresses);\n> +\t size_t aliases_count = array_size (&aliases);\n> +\t char **out_addresses = alloc_buffer_alloc_array\n> +\t (&outbuf, char *, addresses_count + 1);\n> +\t char **out_aliases = alloc_buffer_alloc_array\n> +\t (&outbuf, char *, aliases_count + 1);\n> +\t if (out_addresses == NULL || out_aliases == NULL)\n> +\t {\n> +\t /* The output buffer is not large enough. */\n> +\t *errnop = ERANGE;\n> +\t *herrnop = NETDB_INTERNAL;\n> +\t status = NSS_STATUS_TRYAGAIN;\n> +\t goto out;\n> +\t }\n> +\t memcpy (out_addresses, array_begin (&addresses),\n> +\t\t addresses_count * sizeof (char *));\n> +\t out_addresses[addresses_count] = NULL;\n> +\t memcpy (out_aliases, array_begin (&aliases),\n> +\t\t aliases_count * sizeof (char *));\n> +\t out_aliases[aliases_count] = NULL;\n> +\n> +\t result->h_addr_list = out_addresses;\n> +\t result->h_aliases = out_aliases;\n> +\t}\n> +\n> + status = NSS_STATUS_SUCCESS;\n> + }\n> out:\n> scratch_buffer_free (&tmp_buffer);\n> + array_free (&addresses);\n> + array_free (&aliases);\n> return status;\n> }\n> \n> diff --git a/nss/tst-nss-files-hosts-multi.c b/nss/tst-nss-files-hosts-multi.c\n> new file mode 100644\n> index 0000000000..c86a4e05d0\n> --- /dev/null\n> +++ b/nss/tst-nss-files-hosts-multi.c\n> @@ -0,0 +1,327 @@\n> +/* Parse /etc/hosts in multi mode with many addresses/aliases.\n> + Copyright (C) 2017 Free Software Foundation, Inc.\n> + This file is part of the GNU C Library.\n> +\n> + The GNU C Library is free software; you can redistribute it and/or\n> + modify it under the terms of the GNU Lesser General Public\n> + License as published by the Free Software Foundation; either\n> + version 2.1 of the License, or (at your option) any later version.\n> +\n> + The GNU C Library is distributed in the hope that it will be useful,\n> + but WITHOUT ANY WARRANTY; without even the implied warranty of\n> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\n> + Lesser General Public License for more details.\n> +\n> + You should have received a copy of the GNU Lesser General Public\n> + License along with the GNU C Library; if not, see\n> + . */\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +struct support_chroot *chroot_env;\n> +\n> +static void\n> +prepare (int argc, char **argv)\n> +{\n> + chroot_env = support_chroot_create\n> + ((struct support_chroot_configuration)\n> + {\n> + .resolv_conf = \"\",\n> + .hosts = \"\", /* See write_hosts below. */\n> + .host_conf = \"multi on\\n\",\n> + });\n> +}\n> +\n> +/* Create the /etc/hosts file from outside the chroot. */\n> +static void\n> +write_hosts (int count)\n> +{\n> + TEST_VERIFY (count > 0 && count <= 65535);\n> + FILE *fp = xfopen (chroot_env->path_hosts, \"w\");\n> + fputs (\"127.0.0.1 localhost localhost.localdomain\\n\"\n> + \"::1 localhost localhost.localdomain\\n\",\n> + fp);\n> + for (int i = 0; i < count; ++i)\n> + {\n> + fprintf (fp, \"10.4.%d.%d www4.example.com\\n\",\n> + (i / 256) & 0xff, i & 0xff);\n> + fprintf (fp, \"10.46.%d.%d www.example.com\\n\",\n> + (i / 256) & 0xff, i & 0xff);\n> + fprintf (fp, \"192.0.2.1 alias.example.com v4-%d.example.com\\n\", i);\n> + fprintf (fp, \"2001:db8::6:%x www6.example.com\\n\", i);\n> + fprintf (fp, \"2001:db8::46:%x www.example.com\\n\", i);\n> + fprintf (fp, \"2001:db8::1 alias.example.com v6-%d.example.com\\n\", i);\n> + }\n> + xfclose (fp);\n> +}\n> +\n> +/* Parameters of a single test. */\n> +struct test_params\n> +{\n> + const char *name; /* Name to query. */\n> + const char *marker; /* Address marker for the name. */\n> + int count; /* Number of addresses/aliases. */\n> + int family; /* AF_INET, AF_INET_6 or AF_UNSPEC. */\n> + bool canonname; /* True if AI_CANONNAME should be enabled. */\n> +};\n> +\n> +/* Expected result of gethostbyname/gethostbyname2. */\n> +static char *\n> +expected_ghbn (const struct test_params *params)\n> +{\n> + TEST_VERIFY (params->family == AF_INET || params->family == AF_INET6);\n> +\n> + struct xmemstream expected;\n> + xopen_memstream (&expected);\n> + if (strcmp (params->name, \"alias.example.com\") == 0)\n> + {\n> + fprintf (expected.out, \"name: %s\\n\", params->name);\n> + char af;\n> + if (params->family == AF_INET)\n> + af = '4';\n> + else\n> + af = '6';\n> + for (int i = 0; i < params->count; ++i)\n> + fprintf (expected.out, \"alias: v%c-%d.example.com\\n\", af, i);\n> +\n> + for (int i = 0; i < params->count; ++i)\n> + if (params->family == AF_INET)\n> + fputs (\"address: 192.0.2.1\\n\", expected.out);\n> + else\n> + fputs (\"address: 2001:db8::1\\n\", expected.out);\n> + }\n> + else /* www/www4/www6 name. */\n> + {\n> + bool do_ipv4 = params->family == AF_INET\n> + && strncmp (params->name, \"www6\", 4) != 0;\n> + bool do_ipv6 = params->family == AF_INET6\n> + && strncmp (params->name, \"www4\", 4) != 0;\n> + if (do_ipv4 || do_ipv6)\n> + {\n> + fprintf (expected.out, \"name: %s\\n\", params->name);\n> + if (do_ipv4)\n> + for (int i = 0; i < params->count; ++i)\n> + fprintf (expected.out, \"address: 10.%s.%d.%d\\n\",\n> + params->marker, i / 256, i % 256);\n> + if (do_ipv6)\n> + for (int i = 0; i < params->count; ++i)\n> + fprintf (expected.out, \"address: 2001:db8::%s:%x\\n\",\n> + params->marker, i);\n> + }\n> + else\n> + fputs (\"error: HOST_NOT_FOUND\\n\", expected.out);\n> + }\n> + xfclose_memstream (&expected);\n> + return expected.buffer;\n> +}\n> +\n> +/* Expected result of getaddrinfo. */\n> +static char *\n> +expected_gai (const struct test_params *params)\n> +{\n> + bool do_ipv4 = false;\n> + bool do_ipv6 = false;\n> + if (params->family == AF_UNSPEC)\n> + do_ipv4 = do_ipv6 = true;\n> + else if (params->family == AF_INET)\n> + do_ipv4 = true;\n> + else if (params->family == AF_INET6)\n> + do_ipv6 = true;\n> +\n> + struct xmemstream expected;\n> + xopen_memstream (&expected);\n> + if (strcmp (params->name, \"alias.example.com\") == 0)\n> + {\n> + if (params->canonname)\n> + fprintf (expected.out,\n> + \"flags: AI_CANONNAME\\n\"\n> + \"canonname: %s\\n\",\n> + params->name);\n> +\n> + if (do_ipv4)\n> + for (int i = 0; i < params->count; ++i)\n> + fputs (\"address: STREAM/TCP 192.0.2.1 80\\n\", expected.out);\n> + if (do_ipv6)\n> + for (int i = 0; i < params->count; ++i)\n> + fputs (\"address: STREAM/TCP 2001:db8::1 80\\n\", expected.out);\n> + }\n> + else /* www/www4/www6 name. */\n> + {\n> + if (strncmp (params->name, \"www4\", 4) == 0)\n> + do_ipv6 = false;\n> + else if (strncmp (params->name, \"www6\", 4) == 0)\n> + do_ipv4 = false;\n> + /* Otherwise, we have www as the name, so we do both. */\n> +\n> + if (do_ipv4 || do_ipv6)\n> + {\n> + if (params->canonname)\n> + fprintf (expected.out,\n> + \"flags: AI_CANONNAME\\n\"\n> + \"canonname: %s\\n\",\n> + params->name);\n> +\n> + if (do_ipv4)\n> + for (int i = 0; i < params->count; ++i)\n> + fprintf (expected.out, \"address: STREAM/TCP 10.%s.%d.%d 80\\n\",\n> + params->marker, i / 256, i % 256);\n> + if (do_ipv6)\n> + for (int i = 0; i < params->count; ++i)\n> + fprintf (expected.out,\n> + \"address: STREAM/TCP 2001:db8::%s:%x 80\\n\",\n> + params->marker, i);\n> + }\n> + else\n> + fputs (\"error: Name or service not known\\n\", expected.out);\n> + }\n> + xfclose_memstream (&expected);\n> + return expected.buffer;\n> +}\n> +\n> +static void\n> +run_gbhn_gai (struct test_params *params)\n> +{\n> + char *ctx = xasprintf (\"name=%s marker=%s count=%d family=%d\",\n> + params->name, params->marker, params->count,\n> + params->family);\n> + if (test_verbose > 0)\n> + printf (\"info: %s\\n\", ctx);\n> +\n> + /* Check gethostbyname, gethostbyname2. */\n> + if (params->family == AF_INET)\n> + {\n> + char *expected = expected_ghbn (params);\n> + check_hostent (ctx, gethostbyname (params->name), expected);\n> + free (expected);\n> + }\n> + if (params->family != AF_UNSPEC)\n> + {\n> + char *expected = expected_ghbn (params);\n> + check_hostent (ctx, gethostbyname2 (params->name, params->family),\n> + expected);\n> + free (expected);\n> + }\n> +\n> + /* Check getaddrinfo. */\n> + for (int do_canonical = 0; do_canonical < 2; ++do_canonical)\n> + {\n> + params->canonname = do_canonical;\n> + char *expected = expected_gai (params);\n> + struct addrinfo hints =\n> + {\n> + .ai_family = params->family,\n> + .ai_socktype = SOCK_STREAM,\n> + .ai_protocol = IPPROTO_TCP,\n> + };\n> + if (do_canonical)\n> + hints.ai_flags |= AI_CANONNAME;\n> + struct addrinfo *ai;\n> + int ret = getaddrinfo (params->name, \"80\", &hints, &ai);\n> + check_addrinfo (ctx, ai, ret, expected);\n> + if (ret == 0)\n> + freeaddrinfo (ai);\n> + free (expected);\n> + }\n> +\n> + free (ctx);\n> +}\n> +\n> +/* Callback for the subprocess which runs the test in a chroot. */\n> +static void\n> +subprocess (void *closure)\n> +{\n> + struct test_params *params = closure;\n> +\n> + xchroot (chroot_env->path_chroot);\n> +\n> + static const int families[] = { AF_INET, AF_INET6, AF_UNSPEC, -1 };\n> + static const char *const names[] =\n> + {\n> + \"www.example.com\", \"www4.example.com\", \"www6.example.com\",\n> + \"alias.example.com\",\n> + NULL\n> + };\n> + static const char *const names_marker[] = { \"46\", \"4\", \"6\", \"\" };\n> +\n> + for (int family_idx = 0; families[family_idx] >= 0; ++family_idx)\n> + {\n> + params->family = families[family_idx];\n> + for (int names_idx = 0; names[names_idx] != NULL; ++names_idx)\n> + {\n> + params->name = names[names_idx];\n> + params->marker = names_marker[names_idx];\n> + run_gbhn_gai (params);\n> + }\n> + }\n> +}\n> +\n> +/* Run the test for a specific number of addresses/aliases. */\n> +static void\n> +run_test (int count)\n> +{\n> + write_hosts (count);\n> +\n> + struct test_params params =\n> + {\n> + .count = count,\n> + };\n> +\n> + support_isolate_in_subprocess (subprocess, ¶ms);\n> +}\n> +\n> +static int\n> +do_test (void)\n> +{\n\nI think it is missing the required support to run as a normal use:\n\n[...]\n support_become_root ();\n if (!support_can_chroot ())\n return EXIT_UNSUPPORTED;\n[...]\n\n> + /* This test should not use gigabytes of memory. */\n> + {\n> + struct rlimit limit;\n> + if (getrlimit (RLIMIT_AS, &limit) != 0)\n> + {\n> + printf (\"getrlimit (RLIMIT_AS) failed: %m\\n\");\n> + return 1;\n> + }\n> + long target = 200 * 1024 * 1024;\n> + if (limit.rlim_cur == RLIM_INFINITY || limit.rlim_cur > target)\n> + {\n> + limit.rlim_cur = target;\n> + if (setrlimit (RLIMIT_AS, &limit) != 0)\n> + {\n> + printf (\"setrlimit (RLIMIT_AS) failed: %m\\n\");\n> + return 1;\n> + }\n> + }\n> + }\n> +\n> + __nss_configure_lookup (\"hosts\", \"files\");\n> + if (dlopen (LIBNSS_FILES_SO, RTLD_LAZY) == NULL)\n> + FAIL_EXIT1 (\"could not load \" LIBNSS_DNS_SO \": %s\", dlerror ());\n> +\n> + /* Run the tests with a few different address/alias counts. */\n> + for (int count = 1; count <= 111; ++count)\n> + run_test (count);\n> + run_test (1111);\n> + run_test (22222);\n> +\n> + support_chroot_free (chroot_env);\n> + return 0;\n> +}\n> +\n> +#define PREPARE prepare\n> +#include \n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":["patchwork-incoming@bilbo.ozlabs.org","mailing list libc-alpha@sourceware.org"],"Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=sourceware.org\n\t(client-ip=209.132.180.131; helo=sourceware.org;\n\tenvelope-from=libc-alpha-return-84205-incoming=patchwork.ozlabs.org@sourceware.org;\n\treceiver=)","ozlabs.org; dkim=pass (1024-bit key;\n\tsecure) header.d=sourceware.org header.i=@sourceware.org\n\theader.b=\"dFSG70pI\"; dkim-atps=neutral","sourceware.org; auth=none"],"Received":["from sourceware.org (server1.sourceware.org [209.132.180.131])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xmwX666gMz9sCZ\n\tfor ;\n\tWed, 6 Sep 2017 04:40:54 +1000 (AEST)","(qmail 95491 invoked by alias); 5 Sep 2017 18:40:48 -0000","(qmail 95472 invoked by uid 89); 5 Sep 2017 18:40:47 -0000"],"DomainKey-Signature":"a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id\n\t:list-unsubscribe:list-subscribe:list-archive:list-post\n\t:list-help:sender:subject:to:references:from:message-id:date\n\t:mime-version:in-reply-to:content-type\n\t:content-transfer-encoding; q=dns; s=default; b=VEV54VSQieVeS+CY\n\teG920cWSt4a6CGNb+29/Ju6d83oX9ABzfZ1vZUnDa5NQQao3eA0rjAZCndSAOImS\n\tOTV3U8BMDjNDD+KZZC6tCScK48o6DMvcJLkZPqMNxEzDeoNiqzKCVZIRQHj+X/lI\n\t2Z68iX6ykKqt4kGc58z10J/TYKI=","DKIM-Signature":"v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id\n\t:list-unsubscribe:list-subscribe:list-archive:list-post\n\t:list-help:sender:subject:to:references:from:message-id:date\n\t:mime-version:in-reply-to:content-type\n\t:content-transfer-encoding; s=default; bh=xS8WzZQkuYdACnOTw1F2+o\n\tdKd7s=; b=dFSG70pIzZgSRM0yfEY4nMXW42yG1L8mowTeZgxxeo9YRFgPBxuqnL\n\tXn3u7gGloRO1A5qG0MjT7SJJisghu9EV3G0PkM5vmDJBM9XDbCmTq06lfsjysxwT\n\t8nceCUudyVy+H+KWvXp0Mdjr2JnpQJveRC3vCsbXkgZozaOp0rskA=","Mailing-List":"contact libc-alpha-help@sourceware.org; run by ezmlm","Precedence":"bulk","List-Id":"","List-Unsubscribe":"","List-Subscribe":"","List-Archive":"","List-Post":"","List-Help":",\n\t","Sender":"libc-alpha-owner@sourceware.org","X-Virus-Found":"No","X-Spam-SWARE-Status":"No, score=-27.6 required=5.0 tests=BAYES_00, GIT_PATCH_0,\n\tGIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_LOW,\n\tSPF_PASS autolearn=ham version=3.3.2 spammy=","X-HELO":"mail-qt0-f179.google.com","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:subject:to:references:from:message-id:date\n\t:user-agent:mime-version:in-reply-to:content-language\n\t:content-transfer-encoding;\n\tbh=x8cyFSHLuVeTcQ7Upy8ZFYN/aOTqIUT1HZkU7mleFN0=;\n\tb=EE42kqo9MHBDQEd1sOFUa71A5/cMhOlr31dEk1hRVj5GWgT43GSw6iCDXH8fHuWAXa\n\tI0DwzQD/5DEya0behE/UK+d2/q6oGPQNjyQ09nJWQb0QhDB1DcArFLCxH3Hw+ztrCV2B\n\tnZn3Ewh6GmFB8o1DGVs/kIsjEsTPDumeRMKBEjCvtH8xvuz2jMNdVjJkeSGSuItT97aF\n\tB4569dHYRWIWOPChueZDEQlTZ5Ua/PaTfnh3EPuXUIpkHfSQG2O40TxL7DvPTOQ1B9Zk\n\tV8MmOdCMV12XZG2ratx2990yHWeishsUW8dlIj6wCrMmKUIkTc3/SrwFIhV+Id7h7ite\n\tKs0g==","X-Gm-Message-State":"AHPjjUhxxasKDP63x3agsElRSllhOzVei+WY6kN6BbLe97lp0mRt1CYX\n\t5O26MniLUp1J7XxegF9pOQ==","X-Google-Smtp-Source":"AOwi7QDY4ijPmHkrE9TO7UIqLF9QTInqxB+V5Rcfpk9UQXqDkjw+m9oag5GbP4wXtySmfOnOguPIgQ==","X-Received":"by 10.200.53.58 with SMTP id y55mr22858qtb.83.1504636836953;\n\tTue, 05 Sep 2017 11:40:36 -0700 (PDT)","Subject":"Re: [PATCH] nss_files: Avoid large buffers with many host addresses\n\t[BZ #22078]","To":"libc-alpha@sourceware.org","References":"<20170904173210.3CE87439942E3@oldenburg.str.redhat.com>","From":"Adhemerval Zanella ","Message-ID":"<2ed7620d-87be-74cd-f346-70f08e31a2d0@linaro.org>","Date":"Tue, 5 Sep 2017 15:40:27 -0300","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.2.1","MIME-Version":"1.0","In-Reply-To":"<20170904173210.3CE87439942E3@oldenburg.str.redhat.com>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"8bit"}},{"id":1783743,"web_url":"http://patchwork.ozlabs.org/comment/1783743/","msgid":"<6430d3c4-578c-c6bd-3074-b511223747bf@redhat.com>","list_archive_url":null,"date":"2017-10-10T13:34:26","subject":"Re: [PATCH] nss_files: Avoid large buffers with many host addresses\n\t[BZ #22078]","submitter":{"id":14312,"url":"http://patchwork.ozlabs.org/api/people/14312/","name":"Florian Weimer","email":"fweimer@redhat.com"},"content":"On 09/05/2017 08:40 PM, Adhemerval Zanella wrote:\n>> +/* Type of the address and alias arrays. */\n>> +#define DYNARRAY_STRUCT array\n>> +#define DYNARRAY_ELEMENT char *\n>> +#define DYNARRAY_PREFIX array_\n>> +#include \n>> +\n> This will create a 16 elements vector as default (128 bytes). Should we\n> aim to use the default or can we use a slight large buffer to speed up\n> slightly generic case (assuming generic case use more than 16 entries)?\n\n16 elements is actually too large, I wouldn't expect more than 1 or 2 \nelements in the typical case. But we already have a scratch buffer on \nthe stack, so I don't think it makes sense to override the default size \nto reduce stack usage.\n\nThe rebased patch simplifies the code a bit, by eliminating the new_data \nvariable. We now always copy back the addresses and aliases from the \ndynamic arrays. In addition, this allows us to reuse the space \ninternal_getent allocated to the aliases array.\n\nThanks,\nFlorian\nThe previous implementation had at least a quadratic space\nrequirement in the number of host addresses and aliases.\n\n2017-10-10 Florian Weimer \n\n\t[BZ #22078]\n\tAvoid large NSS buffers with many addresses, aliases.\n\t* nss/nss_files/files-hosts.c (gethostbyname3_multi): Rewrite\n\tusing dynarrays and struct alloc_buffer.\n\t* nss/Makefile (tests): Add tst-nss-files-hosts-multi.\n\t(tst-nss-files-hosts-multi): Link with -ldl.\n\t* nss/tst-nss-files-hosts-multi.c: New file.\n\ndiff --git a/nss/Makefile b/nss/Makefile\nindex f27bed11fc..26952112c1 100644\n--- a/nss/Makefile\n+++ b/nss/Makefile\n@@ -63,6 +63,7 @@ xtests\t\t\t= bug-erange\n # Tests which need libdl\n ifeq (yes,$(build-shared))\n tests += tst-nss-files-hosts-erange\n+tests += tst-nss-files-hosts-multi\n endif\n \n # If we have a thread library then we can test cancellation against\n@@ -167,3 +168,4 @@ $(objpfx)tst-cancel-getpwuid_r: $(shared-thread-library)\n endif\n \n $(objpfx)tst-nss-files-hosts-erange: $(libdl)\n+$(objpfx)tst-nss-files-hosts-multi: $(libdl)\ndiff --git a/nss/nss_files/files-hosts.c b/nss/nss_files/files-hosts.c\nindex 763fa39a47..6f7cc4d94b 100644\n--- a/nss/nss_files/files-hosts.c\n+++ b/nss/nss_files/files-hosts.c\n@@ -23,6 +23,7 @@\n #include \n #include \n #include \n+#include \n \n \n /* Get implementation for some internal functions. */\n@@ -116,24 +117,45 @@ DB_LOOKUP (hostbyaddr, ,,,\n \t }, const void *addr, socklen_t len, int af)\n #undef EXTRA_ARGS_VALUE\n \n+/* Type of the address and alias arrays. */\n+#define DYNARRAY_STRUCT array\n+#define DYNARRAY_ELEMENT char *\n+#define DYNARRAY_PREFIX array_\n+#include \n+\n static enum nss_status\n gethostbyname3_multi (FILE * stream, const char *name, int af,\n \t\t struct hostent *result, char *buffer, size_t buflen,\n \t\t int *errnop, int *herrnop, int flags)\n {\n+ assert (af == AF_INET || af == AF_INET6);\n+\n /* We have to get all host entries from the file. */\n struct scratch_buffer tmp_buffer;\n scratch_buffer_init (&tmp_buffer);\n struct hostent tmp_result_buf;\n- int naddrs = 1;\n- int naliases = 0;\n- char *bufferend;\n+ struct array addresses;\n+ array_init (&addresses);\n+ struct array aliases;\n+ array_init (&aliases);\n enum nss_status status;\n \n- while (result->h_aliases[naliases] != NULL)\n- ++naliases;\n+ /* Preserve the addresses and aliases encountered so far. */\n+ for (size_t i = 0; result->h_addr_list[i] != NULL; ++i)\n+ array_add (&addresses, result->h_addr_list[i]);\n+ for (size_t i = 0; result->h_aliases[i] != NULL; ++i)\n+ array_add (&aliases, result->h_aliases[i]);\n \n- bufferend = (char *) &result->h_aliases[naliases + 1];\n+ /* The output buffer re-uses now-unused space at the end of the\n+ buffer, starting with the aliases array. It comes last in the\n+ data produced by internal_getent. (The alias names themselves\n+ are still located in the line read in internal_getent, which is\n+ stored at the beginning of the buffer.) */\n+ struct alloc_buffer outbuf;\n+ {\n+ char *bufferend = (char *) result->h_aliases;\n+ outbuf = alloc_buffer_create (bufferend, buffer + buflen - bufferend);\n+ }\n \n while (true)\n {\n@@ -170,46 +192,74 @@ gethostbyname3_multi (FILE * stream, const char *name, int af,\n \t }\n \t while ((matches = 0));\n \n+\t /* If the line matches, we need to copy the addresses and\n+\t aliases, so that we can reuse tmp_buffer for the next\n+\t line. */\n \t if (matches)\n \t {\n-\t /* We could be very clever and try to recycle a few bytes\n-\t\t in the buffer instead of generating new arrays. But\n-\t\t we are not doing this here since it's more work than\n-\t\t it's worth. Simply let the user provide a bit bigger\n-\t\t buffer. */\n-\t char **new_h_addr_list;\n-\t char **new_h_aliases;\n-\t int newaliases = 0;\n-\t size_t newstrlen = 0;\n-\t int cnt;\n+\t /* Record the addresses. */\n+\t for (size_t i = 0; tmp_result_buf.h_addr_list[i] != NULL; ++i)\n+\t\t{\n+\t\t /* Allocate the target space in the output buffer,\n+\t\t depending on the address family. */\n+\t\t void *target;\n+\t\t if (af == AF_INET)\n+\t\t {\n+\t\t assert (tmp_result_buf.h_length == 4);\n+\t\t target = alloc_buffer_alloc (&outbuf, struct in_addr);\n+\t\t }\n+\t\t else if (af == AF_INET6)\n+\t\t {\n+\t\t assert (tmp_result_buf.h_length == 16);\n+\t\t target = alloc_buffer_alloc (&outbuf, struct in6_addr);\n+\t\t }\n+\t\t else\n+\t\t __builtin_unreachable ();\n+\n+\t\t if (target == NULL)\n+\t\t {\n+\t\t /* Request a larger output buffer. */\n+\t\t *errnop = ERANGE;\n+\t\t *herrnop = NETDB_INTERNAL;\n+\t\t status = NSS_STATUS_TRYAGAIN;\n+\t\t break;\n+\t\t }\n+\t\t memcpy (target, tmp_result_buf.h_addr_list[i],\n+\t\t\t tmp_result_buf.h_length);\n+\t\t array_add (&addresses, target);\n+\t\t}\n \n-\t /* Count the new aliases and the length of the strings. */\n-\t while (tmp_result_buf.h_aliases[newaliases] != NULL)\n+\t /* Record the aliases. */\n+\t for (size_t i = 0; tmp_result_buf.h_aliases[i] != NULL; ++i)\n \t\t{\n-\t\t char *cp = tmp_result_buf.h_aliases[newaliases];\n-\t\t ++newaliases;\n-\t\t newstrlen += strlen (cp) + 1;\n+\t\t char *alias = tmp_result_buf.h_aliases[i];\n+\t\t array_add (&aliases,\n+\t\t\t alloc_buffer_copy_string (&outbuf, alias));\n \t\t}\n-\t /* If the real name is different add it also to the\n-\t\t aliases. This means that there is a duplication\n-\t\t in the alias list but this is really the user's\n+\n+\t /* If the real name is different add, it also to the\n+\t\t aliases. This means that there is a duplication in\n+\t\t the alias list but this is really the user's\n \t\t problem. */\n-\t if (strcmp (old_result->h_name,\n-\t\t\t tmp_result_buf.h_name) != 0)\n+\t {\n+\t\tchar *new_name = tmp_result_buf.h_name;\n+\t\tif (strcmp (old_result->h_name, new_name) != 0)\n+\t\t array_add (&aliases,\n+\t\t\t alloc_buffer_copy_string (&outbuf, new_name));\n+\t }\n+\n+\t /* Report memory allocation failures during the\n+\t\t expansion of the temporary arrays. */\n+\t if (array_has_failed (&addresses) || array_has_failed (&aliases))\n \t\t{\n-\t\t ++newaliases;\n-\t\t newstrlen += strlen (tmp_result_buf.h_name) + 1;\n+\t\t *errnop = ENOMEM;\n+\t\t *herrnop = NETDB_INTERNAL;\n+\t\t status = NSS_STATUS_UNAVAIL;\n+\t\t break;\n \t\t}\n \n-\t /* Make sure bufferend is aligned. */\n-\t assert ((bufferend - (char *) 0) % sizeof (char *) == 0);\n-\n-\t /* Now we can check whether the buffer is large enough.\n-\t\t 16 is the maximal size of the IP address. */\n-\t if (bufferend + 16 + (naddrs + 2) * sizeof (char *)\n-\t\t + roundup (newstrlen, sizeof (char *))\n-\t\t + (naliases + newaliases + 1) * sizeof (char *)\n-\t\t >= buffer + buflen)\n+\t /* Request a larger output buffer if we ran out of room. */\n+\t if (alloc_buffer_has_failed (&outbuf))\n \t\t{\n \t\t *errnop = ERANGE;\n \t\t *herrnop = NETDB_INTERNAL;\n@@ -217,63 +267,6 @@ gethostbyname3_multi (FILE * stream, const char *name, int af,\n \t\t break;\n \t\t}\n \n-\t new_h_addr_list =\n-\t\t(char **) (bufferend\n-\t\t\t + roundup (newstrlen, sizeof (char *))\n-\t\t\t + 16);\n-\t new_h_aliases =\n-\t\t(char **) ((char *) new_h_addr_list\n-\t\t\t + (naddrs + 2) * sizeof (char *));\n-\n-\t /* Copy the old data in the new arrays. */\n-\t for (cnt = 0; cnt < naddrs; ++cnt)\n-\t\tnew_h_addr_list[cnt] = old_result->h_addr_list[cnt];\n-\n-\t for (cnt = 0; cnt < naliases; ++cnt)\n-\t\tnew_h_aliases[cnt] = old_result->h_aliases[cnt];\n-\n-\t /* Store the new strings. */\n-\t cnt = 0;\n-\t while (tmp_result_buf.h_aliases[cnt] != NULL)\n-\t\t{\n-\t\t new_h_aliases[naliases++] = bufferend;\n-\t\t bufferend = (__stpcpy (bufferend,\n-\t\t\t\t\t tmp_result_buf.h_aliases[cnt])\n-\t\t\t + 1);\n-\t\t ++cnt;\n-\t\t}\n-\n-\t if (cnt < newaliases)\n-\t\t{\n-\t\t new_h_aliases[naliases++] = bufferend;\n-\t\t bufferend = __stpcpy (bufferend,\n-\t\t\t\t\ttmp_result_buf.h_name) + 1;\n-\t\t}\n-\n-\t /* Final NULL pointer. */\n-\t new_h_aliases[naliases] = NULL;\n-\n-\t /* Round up the buffer end address. */\n-\t bufferend += (sizeof (char *)\n-\t\t\t - ((bufferend - (char *) 0)\n-\t\t\t % sizeof (char *))) % sizeof (char *);\n-\n-\t /* Now the new address. */\n-\t new_h_addr_list[naddrs++] =\n-\t\tmemcpy (bufferend, tmp_result_buf.h_addr,\n-\t\t\ttmp_result_buf.h_length);\n-\n-\t /* Also here a final NULL pointer. */\n-\t new_h_addr_list[naddrs] = NULL;\n-\n-\t /* Store the new array pointers. */\n-\t old_result->h_aliases = new_h_aliases;\n-\t old_result->h_addr_list = new_h_addr_list;\n-\n-\t /* Compute the new buffer end. */\n-\t bufferend = (char *) &new_h_aliases[naliases + 1];\n-\t assert (bufferend <= buffer + buflen);\n-\n \t result = old_result;\n \t } /* If match was found. */\n \n@@ -293,7 +286,47 @@ gethostbyname3_multi (FILE * stream, const char *name, int af,\n if (status != NSS_STATUS_TRYAGAIN)\n status = NSS_STATUS_SUCCESS;\n \n+ if (status == NSS_STATUS_SUCCESS)\n+ {\n+ /* Copy the address and alias arrays into the output buffer and\n+\t add NULL terminators. The pointed-to elements were directly\n+\t written into the output buffer above and do not need to be\n+\t copied again. */\n+ size_t addresses_count = array_size (&addresses);\n+ size_t aliases_count = array_size (&aliases);\n+ char **out_addresses = alloc_buffer_alloc_array\n+\t(&outbuf, char *, addresses_count + 1);\n+ char **out_aliases = alloc_buffer_alloc_array\n+\t(&outbuf, char *, aliases_count + 1);\n+ if (out_addresses == NULL || out_aliases == NULL)\n+\t{\n+\t /* The output buffer is not large enough. */\n+\t *errnop = ERANGE;\n+\t *herrnop = NETDB_INTERNAL;\n+\t status = NSS_STATUS_TRYAGAIN;\n+\t /* Fall through to function exit. */\n+\t}\n+ else\n+\t{\n+\t /* Everything is allocated in place. Make the copies and\n+\t adjust the array pointers. */\n+\t memcpy (out_addresses, array_begin (&addresses),\n+\t\t addresses_count * sizeof (char *));\n+\t out_addresses[addresses_count] = NULL;\n+\t memcpy (out_aliases, array_begin (&aliases),\n+\t\t aliases_count * sizeof (char *));\n+\t out_aliases[aliases_count] = NULL;\n+\n+\t result->h_addr_list = out_addresses;\n+\t result->h_aliases = out_aliases;\n+\n+\t status = NSS_STATUS_SUCCESS;\n+\t}\n+ }\n+\n scratch_buffer_free (&tmp_buffer);\n+ array_free (&addresses);\n+ array_free (&aliases);\n return status;\n }\n \ndiff --git a/nss/tst-nss-files-hosts-multi.c b/nss/tst-nss-files-hosts-multi.c\nnew file mode 100644\nindex 0000000000..195a19be4f\n--- /dev/null\n+++ b/nss/tst-nss-files-hosts-multi.c\n@@ -0,0 +1,331 @@\n+/* Parse /etc/hosts in multi mode with many addresses/aliases.\n+ Copyright (C) 2017 Free Software Foundation, Inc.\n+ This file is part of the GNU C Library.\n+\n+ The GNU C Library is free software; you can redistribute it and/or\n+ modify it under the terms of the GNU Lesser General Public\n+ License as published by the Free Software Foundation; either\n+ version 2.1 of the License, or (at your option) any later version.\n+\n+ The GNU C Library is distributed in the hope that it will be useful,\n+ but WITHOUT ANY WARRANTY; without even the implied warranty of\n+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\n+ Lesser General Public License for more details.\n+\n+ You should have received a copy of the GNU Lesser General Public\n+ License along with the GNU C Library; if not, see\n+ . */\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+struct support_chroot *chroot_env;\n+\n+static void\n+prepare (int argc, char **argv)\n+{\n+ chroot_env = support_chroot_create\n+ ((struct support_chroot_configuration)\n+ {\n+ .resolv_conf = \"\",\n+ .hosts = \"\", /* See write_hosts below. */\n+ .host_conf = \"multi on\\n\",\n+ });\n+}\n+\n+/* Create the /etc/hosts file from outside the chroot. */\n+static void\n+write_hosts (int count)\n+{\n+ TEST_VERIFY (count > 0 && count <= 65535);\n+ FILE *fp = xfopen (chroot_env->path_hosts, \"w\");\n+ fputs (\"127.0.0.1 localhost localhost.localdomain\\n\"\n+ \"::1 localhost localhost.localdomain\\n\",\n+ fp);\n+ for (int i = 0; i < count; ++i)\n+ {\n+ fprintf (fp, \"10.4.%d.%d www4.example.com\\n\",\n+ (i / 256) & 0xff, i & 0xff);\n+ fprintf (fp, \"10.46.%d.%d www.example.com\\n\",\n+ (i / 256) & 0xff, i & 0xff);\n+ fprintf (fp, \"192.0.2.1 alias.example.com v4-%d.example.com\\n\", i);\n+ fprintf (fp, \"2001:db8::6:%x www6.example.com\\n\", i);\n+ fprintf (fp, \"2001:db8::46:%x www.example.com\\n\", i);\n+ fprintf (fp, \"2001:db8::1 alias.example.com v6-%d.example.com\\n\", i);\n+ }\n+ xfclose (fp);\n+}\n+\n+/* Parameters of a single test. */\n+struct test_params\n+{\n+ const char *name; /* Name to query. */\n+ const char *marker; /* Address marker for the name. */\n+ int count; /* Number of addresses/aliases. */\n+ int family; /* AF_INET, AF_INET_6 or AF_UNSPEC. */\n+ bool canonname; /* True if AI_CANONNAME should be enabled. */\n+};\n+\n+/* Expected result of gethostbyname/gethostbyname2. */\n+static char *\n+expected_ghbn (const struct test_params *params)\n+{\n+ TEST_VERIFY (params->family == AF_INET || params->family == AF_INET6);\n+\n+ struct xmemstream expected;\n+ xopen_memstream (&expected);\n+ if (strcmp (params->name, \"alias.example.com\") == 0)\n+ {\n+ fprintf (expected.out, \"name: %s\\n\", params->name);\n+ char af;\n+ if (params->family == AF_INET)\n+ af = '4';\n+ else\n+ af = '6';\n+ for (int i = 0; i < params->count; ++i)\n+ fprintf (expected.out, \"alias: v%c-%d.example.com\\n\", af, i);\n+\n+ for (int i = 0; i < params->count; ++i)\n+ if (params->family == AF_INET)\n+ fputs (\"address: 192.0.2.1\\n\", expected.out);\n+ else\n+ fputs (\"address: 2001:db8::1\\n\", expected.out);\n+ }\n+ else /* www/www4/www6 name. */\n+ {\n+ bool do_ipv4 = params->family == AF_INET\n+ && strncmp (params->name, \"www6\", 4) != 0;\n+ bool do_ipv6 = params->family == AF_INET6\n+ && strncmp (params->name, \"www4\", 4) != 0;\n+ if (do_ipv4 || do_ipv6)\n+ {\n+ fprintf (expected.out, \"name: %s\\n\", params->name);\n+ if (do_ipv4)\n+ for (int i = 0; i < params->count; ++i)\n+ fprintf (expected.out, \"address: 10.%s.%d.%d\\n\",\n+ params->marker, i / 256, i % 256);\n+ if (do_ipv6)\n+ for (int i = 0; i < params->count; ++i)\n+ fprintf (expected.out, \"address: 2001:db8::%s:%x\\n\",\n+ params->marker, i);\n+ }\n+ else\n+ fputs (\"error: HOST_NOT_FOUND\\n\", expected.out);\n+ }\n+ xfclose_memstream (&expected);\n+ return expected.buffer;\n+}\n+\n+/* Expected result of getaddrinfo. */\n+static char *\n+expected_gai (const struct test_params *params)\n+{\n+ bool do_ipv4 = false;\n+ bool do_ipv6 = false;\n+ if (params->family == AF_UNSPEC)\n+ do_ipv4 = do_ipv6 = true;\n+ else if (params->family == AF_INET)\n+ do_ipv4 = true;\n+ else if (params->family == AF_INET6)\n+ do_ipv6 = true;\n+\n+ struct xmemstream expected;\n+ xopen_memstream (&expected);\n+ if (strcmp (params->name, \"alias.example.com\") == 0)\n+ {\n+ if (params->canonname)\n+ fprintf (expected.out,\n+ \"flags: AI_CANONNAME\\n\"\n+ \"canonname: %s\\n\",\n+ params->name);\n+\n+ if (do_ipv4)\n+ for (int i = 0; i < params->count; ++i)\n+ fputs (\"address: STREAM/TCP 192.0.2.1 80\\n\", expected.out);\n+ if (do_ipv6)\n+ for (int i = 0; i < params->count; ++i)\n+ fputs (\"address: STREAM/TCP 2001:db8::1 80\\n\", expected.out);\n+ }\n+ else /* www/www4/www6 name. */\n+ {\n+ if (strncmp (params->name, \"www4\", 4) == 0)\n+ do_ipv6 = false;\n+ else if (strncmp (params->name, \"www6\", 4) == 0)\n+ do_ipv4 = false;\n+ /* Otherwise, we have www as the name, so we do both. */\n+\n+ if (do_ipv4 || do_ipv6)\n+ {\n+ if (params->canonname)\n+ fprintf (expected.out,\n+ \"flags: AI_CANONNAME\\n\"\n+ \"canonname: %s\\n\",\n+ params->name);\n+\n+ if (do_ipv4)\n+ for (int i = 0; i < params->count; ++i)\n+ fprintf (expected.out, \"address: STREAM/TCP 10.%s.%d.%d 80\\n\",\n+ params->marker, i / 256, i % 256);\n+ if (do_ipv6)\n+ for (int i = 0; i < params->count; ++i)\n+ fprintf (expected.out,\n+ \"address: STREAM/TCP 2001:db8::%s:%x 80\\n\",\n+ params->marker, i);\n+ }\n+ else\n+ fputs (\"error: Name or service not known\\n\", expected.out);\n+ }\n+ xfclose_memstream (&expected);\n+ return expected.buffer;\n+}\n+\n+static void\n+run_gbhn_gai (struct test_params *params)\n+{\n+ char *ctx = xasprintf (\"name=%s marker=%s count=%d family=%d\",\n+ params->name, params->marker, params->count,\n+ params->family);\n+ if (test_verbose > 0)\n+ printf (\"info: %s\\n\", ctx);\n+\n+ /* Check gethostbyname, gethostbyname2. */\n+ if (params->family == AF_INET)\n+ {\n+ char *expected = expected_ghbn (params);\n+ check_hostent (ctx, gethostbyname (params->name), expected);\n+ free (expected);\n+ }\n+ if (params->family != AF_UNSPEC)\n+ {\n+ char *expected = expected_ghbn (params);\n+ check_hostent (ctx, gethostbyname2 (params->name, params->family),\n+ expected);\n+ free (expected);\n+ }\n+\n+ /* Check getaddrinfo. */\n+ for (int do_canonical = 0; do_canonical < 2; ++do_canonical)\n+ {\n+ params->canonname = do_canonical;\n+ char *expected = expected_gai (params);\n+ struct addrinfo hints =\n+ {\n+ .ai_family = params->family,\n+ .ai_socktype = SOCK_STREAM,\n+ .ai_protocol = IPPROTO_TCP,\n+ };\n+ if (do_canonical)\n+ hints.ai_flags |= AI_CANONNAME;\n+ struct addrinfo *ai;\n+ int ret = getaddrinfo (params->name, \"80\", &hints, &ai);\n+ check_addrinfo (ctx, ai, ret, expected);\n+ if (ret == 0)\n+ freeaddrinfo (ai);\n+ free (expected);\n+ }\n+\n+ free (ctx);\n+}\n+\n+/* Callback for the subprocess which runs the test in a chroot. */\n+static void\n+subprocess (void *closure)\n+{\n+ struct test_params *params = closure;\n+\n+ xchroot (chroot_env->path_chroot);\n+\n+ static const int families[] = { AF_INET, AF_INET6, AF_UNSPEC, -1 };\n+ static const char *const names[] =\n+ {\n+ \"www.example.com\", \"www4.example.com\", \"www6.example.com\",\n+ \"alias.example.com\",\n+ NULL\n+ };\n+ static const char *const names_marker[] = { \"46\", \"4\", \"6\", \"\" };\n+\n+ for (int family_idx = 0; families[family_idx] >= 0; ++family_idx)\n+ {\n+ params->family = families[family_idx];\n+ for (int names_idx = 0; names[names_idx] != NULL; ++names_idx)\n+ {\n+ params->name = names[names_idx];\n+ params->marker = names_marker[names_idx];\n+ run_gbhn_gai (params);\n+ }\n+ }\n+}\n+\n+/* Run the test for a specific number of addresses/aliases. */\n+static void\n+run_test (int count)\n+{\n+ write_hosts (count);\n+\n+ struct test_params params =\n+ {\n+ .count = count,\n+ };\n+\n+ support_isolate_in_subprocess (subprocess, ¶ms);\n+}\n+\n+static int\n+do_test (void)\n+{\n+ support_become_root ();\n+ if (!support_can_chroot ())\n+ return EXIT_UNSUPPORTED;\n+\n+ /* This test should not use gigabytes of memory. */\n+ {\n+ struct rlimit limit;\n+ if (getrlimit (RLIMIT_AS, &limit) != 0)\n+ {\n+ printf (\"getrlimit (RLIMIT_AS) failed: %m\\n\");\n+ return 1;\n+ }\n+ long target = 200 * 1024 * 1024;\n+ if (limit.rlim_cur == RLIM_INFINITY || limit.rlim_cur > target)\n+ {\n+ limit.rlim_cur = target;\n+ if (setrlimit (RLIMIT_AS, &limit) != 0)\n+ {\n+ printf (\"setrlimit (RLIMIT_AS) failed: %m\\n\");\n+ return 1;\n+ }\n+ }\n+ }\n+\n+ __nss_configure_lookup (\"hosts\", \"files\");\n+ if (dlopen (LIBNSS_FILES_SO, RTLD_LAZY) == NULL)\n+ FAIL_EXIT1 (\"could not load \" LIBNSS_DNS_SO \": %s\", dlerror ());\n+\n+ /* Run the tests with a few different address/alias counts. */\n+ for (int count = 1; count <= 111; ++count)\n+ run_test (count);\n+ run_test (1111);\n+ run_test (22222);\n+\n+ support_chroot_free (chroot_env);\n+ return 0;\n+}\n+\n+#define PREPARE prepare\n+#include ","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":["patchwork-incoming@bilbo.ozlabs.org","mailing list libc-alpha@sourceware.org"],"Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=sourceware.org\n\t(client-ip=209.132.180.131; helo=sourceware.org;\n\tenvelope-from=libc-alpha-return-85577-incoming=patchwork.ozlabs.org@sourceware.org;\n\treceiver=)","ozlabs.org; dkim=pass (1024-bit key;\n\tsecure) header.d=sourceware.org header.i=@sourceware.org\n\theader.b=\"PTgMsUvq\"; dkim-atps=neutral","sourceware.org; auth=none","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=fweimer@redhat.com"],"Received":["from sourceware.org (server1.sourceware.org [209.132.180.131])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yBJKq6Vwwz9tYS\n\tfor ;\n\tWed, 11 Oct 2017 00:46:07 +1100 (AEDT)","(qmail 100360 invoked by alias); 10 Oct 2017 13:34:36 -0000","(qmail 99768 invoked by uid 89); 10 Oct 2017 13:34:35 -0000"],"DomainKey-Signature":"a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id\n\t:list-unsubscribe:list-subscribe:list-archive:list-post\n\t:list-help:sender:subject:to:references:from:message-id:date\n\t:mime-version:in-reply-to:content-type; q=dns; s=default; b=KF24\n\tlr8zyziB9K1KgjTo38JDDwT75ZAUv5kb/Zsx6M5VQRjEOjieA5k+3EjbL6j/sl3R\n\t+AAn94UiRpai9pYlPwNOpoHY7WgTo68T9ZEeADsm7p1YtXcJh5nEr4wgYUfBtLtc\n\tmhBtcGIvP/M48IRNnZ8IXDnF2wIYNmI4L69mx30=","DKIM-Signature":"v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id\n\t:list-unsubscribe:list-subscribe:list-archive:list-post\n\t:list-help:sender:subject:to:references:from:message-id:date\n\t:mime-version:in-reply-to:content-type; s=default; bh=RLnnymOtNM\n\tbchb1WApxHUVGreJI=; b=PTgMsUvqSomgQs3xM1wAeXQkTlsJEvSbVtkLJYtUos\n\tBM/JKTLYZGYKp4ETysf/3qmrWwRFHAAAgtS8DqVHGoduVDFzke9QvgIIkv7kkeW6\n\tWVRKiLwlM5VcOmTLQM1BBFXJWY/Dttahb1kGnrFzZXa/v9u8EjfLnrmdeA4+MkJh\n\tQ=","Mailing-List":"contact libc-alpha-help@sourceware.org; run by ezmlm","Precedence":"bulk","List-Id":"","List-Unsubscribe":"","List-Subscribe":"","List-Archive":"","List-Post":"","List-Help":",\n\t","Sender":"libc-alpha-owner@sourceware.org","X-Virus-Found":"No","X-Spam-SWARE-Status":"No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0,\n\tGIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RP_MATCHES_RCVD,\n\tSPF_HELO_PASS autolearn=ham version=3.3.2 spammy=cancellation","X-HELO":"mx1.redhat.com","DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com AB3FAC04AC6B","Subject":"Re: [PATCH] nss_files: Avoid large buffers with many host addresses\n\t[BZ #22078]","To":"Adhemerval Zanella ,\n\tlibc-alpha@sourceware.org","References":"<20170904173210.3CE87439942E3@oldenburg.str.redhat.com>\n\t<2ed7620d-87be-74cd-f346-70f08e31a2d0@linaro.org>","From":"Florian Weimer ","Message-ID":"<6430d3c4-578c-c6bd-3074-b511223747bf@redhat.com>","Date":"Tue, 10 Oct 2017 15:34:26 +0200","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<2ed7620d-87be-74cd-f346-70f08e31a2d0@linaro.org>","Content-Type":"multipart/mixed;\n\tboundary=\"------------E04F9A32A79773D9E01A13AC\""}},{"id":1784033,"web_url":"http://patchwork.ozlabs.org/comment/1784033/","msgid":"","list_archive_url":null,"date":"2017-10-10T18:41:11","subject":"Re: [PATCH] nss_files: Avoid large buffers with many host addresses\n\t[BZ #22078]","submitter":{"id":66065,"url":"http://patchwork.ozlabs.org/api/people/66065/","name":"Adhemerval Zanella Netto","email":"adhemerval.zanella@linaro.org"},"content":"On 10/10/2017 10:34, Florian Weimer wrote:\n> On 09/05/2017 08:40 PM, Adhemerval Zanella wrote:\n>>> +/* Type of the address and alias arrays.  */\n>>> +#define DYNARRAY_STRUCT array\n>>> +#define DYNARRAY_ELEMENT char *\n>>> +#define DYNARRAY_PREFIX array_\n>>> +#include \n>>> +\n>> This will create a 16 elements vector as default (128 bytes). Should we\n>> aim to use the default or can we use a slight large buffer to speed up\n>> slightly generic case (assuming generic case use more than 16 entries)?\n> \n> 16 elements is actually too large, I wouldn't expect more than 1 or 2 elements in the typical case.  But we already have a scratch buffer on the stack, so I don't think it makes sense to override the default size to reduce stack usage.\n> \n> The rebased patch simplifies the code a bit, by eliminating the new_data variable.  We now always copy back the addresses and aliases from the dynamic arrays.  In addition, this allows us to reuse the space internal_getent allocated to the aliases array.\n> \n> Thanks,\n> Florian\n> \n> bug22078.patch\n\nLGTM.","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":["patchwork-incoming@bilbo.ozlabs.org","mailing list libc-alpha@sourceware.org"],"Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=sourceware.org\n\t(client-ip=209.132.180.131; helo=sourceware.org;\n\tenvelope-from=libc-alpha-return-85613-incoming=patchwork.ozlabs.org@sourceware.org;\n\treceiver=)","ozlabs.org; dkim=pass (1024-bit key;\n\tsecure) header.d=sourceware.org header.i=@sourceware.org\n\theader.b=\"xWpzUa/I\"; dkim-atps=neutral","sourceware.org; auth=none"],"Received":["from sourceware.org (server1.sourceware.org [209.132.180.131])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yBQxb2kLlz9t69\n\tfor ;\n\tWed, 11 Oct 2017 05:44:03 +1100 (AEDT)","(qmail 84660 invoked by alias); 10 Oct 2017 18:41:19 -0000","(qmail 84478 invoked by uid 89); 10 Oct 2017 18:41:19 -0000"],"DomainKey-Signature":"a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id\n\t:list-unsubscribe:list-subscribe:list-archive:list-post\n\t:list-help:sender:subject:to:references:from:message-id:date\n\t:mime-version:in-reply-to:content-type\n\t:content-transfer-encoding; q=dns; s=default; b=eEHKlzs/LRb4lgxE\n\tsawxcvmO9U61gEKS5M/LuOH5T0oOIArVKDjx3yrNgWMS4JpYPFcxPyL+3DDTBzNw\n\tzFRa3A2344zDTFNFV6l+4orWyyzTFkaZQQSpreQRE9aHPzPwMNZphktPHZIWfU9r\n\t3ty8qZxP3SzzYRAuFmoopgj3CYk=","DKIM-Signature":"v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id\n\t:list-unsubscribe:list-subscribe:list-archive:list-post\n\t:list-help:sender:subject:to:references:from:message-id:date\n\t:mime-version:in-reply-to:content-type\n\t:content-transfer-encoding; s=default; bh=1taefTAHi4a1l1T0l3Xs/e\n\tTvxL0=; b=xWpzUa/IHHNtAlilPuNnW8gkZU22BPWobcnn2Oh+QMHHdgLAXHDyn2\n\t7kQPnb98+t5EMP+954GtOTv2enIEBVJjicI1IhtFjFRC+Ey7IO+wz6gKiWEx5Dna\n\tuiiPcXlMa+8+NnDfSJ8n/yHiEbQm+mcWa4VzEO+rv4nsul0W8z3Ag=","Mailing-List":"contact libc-alpha-help@sourceware.org; run by ezmlm","Precedence":"bulk","List-Id":"","List-Unsubscribe":"","List-Subscribe":"","List-Archive":"","List-Post":"","List-Help":",\n\t","Sender":"libc-alpha-owner@sourceware.org","X-Virus-Found":"No","X-Spam-SWARE-Status":"No, score=-1.4 required=5.0 tests=BAYES_00,\n\tRCVD_IN_DNSWL_NONE, RCVD_IN_SORBS_SPAM,\n\tSPF_PASS autolearn=no version=3.3.2\n\tspammy=HContent-Transfer-Encoding:8bit","X-HELO":"mail-qt0-f169.google.com","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:subject:to:references:from:message-id:date\n\t:user-agent:mime-version:in-reply-to:content-language\n\t:content-transfer-encoding;\n\tbh=J49xXbcgidRtSdqA6w+lk/6hY4/Obobs6xOkpbJb3UQ=;\n\tb=N+0fxHxv1hixY+ggBq39EzhsELcC21UnwKbwzhwkZ8EnInJR+slaYrbjCcL8Cc1TcG\n\tpk0FUf3T9fGTRlDtn9hes69xiqykopWV6kpccRK3BOGBCIdgBnI84cdwjjhXfw58XXx6\n\t5xLrWu0lg3sKUkIhmVD6Of5Cq13pWDd66aupbSzKeSHcgg6X6O9DJEbcq0a579JudKcE\n\tX0GG9KPdCddog+2Yx4gAe/OKULCCbaxDjy0804q975advKFawUyAcmfSpStXxrp2TBlP\n\t1Fhekhm7ZGnOAOhuXaP7Esolnt7nMnIuD1QyggzKWsd7PO4q6g0H/9OHY5YU6EOwyvLG\n\tyPgw==","X-Gm-Message-State":"AMCzsaXuVaiPUmNHep/4ePXH1tcGdcPUF3auLopWuJPUSroOQ4qv57ii\n\to1l4mkVzGmGIAWt/pUtEWFC9POyK8Oo=","X-Google-Smtp-Source":"AOwi7QAelX0ip9V7dQYbH0ELRGeURFx/jNz73P8XgBrCZdijdFuK142jQSqZW2ypVRIiD2SbuaMOGg==","X-Received":"by 10.200.15.250 with SMTP id f55mr17704306qtk.249.1507660876028;\n\tTue, 10 Oct 2017 11:41:16 -0700 (PDT)","Subject":"Re: [PATCH] nss_files: Avoid large buffers with many host addresses\n\t[BZ #22078]","To":"Florian Weimer , libc-alpha@sourceware.org","References":"<20170904173210.3CE87439942E3@oldenburg.str.redhat.com>\n\t<2ed7620d-87be-74cd-f346-70f08e31a2d0@linaro.org>\n\t<6430d3c4-578c-c6bd-3074-b511223747bf@redhat.com>","From":"Adhemerval Zanella ","Message-ID":"","Date":"Tue, 10 Oct 2017 15:41:11 -0300","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<6430d3c4-578c-c6bd-3074-b511223747bf@redhat.com>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"8bit"}}]