php: fpm sapi: install startup script

When PHP's FastCGI Process Manager SAPI is selected:

- install a startup script.
- install a simple configuration, using the ondemand process manager
  that only starts children when the website is actually being used.

Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>
---
 package/php/php-fpm.conf | 14 ++++++++++++++
 package/php/php.mk       | 15 +++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 package/php/php-fpm.conf

On 04/30/15 21:36, Floris Bos wrote:
> When PHP's FastCGI Process Manager SAPI is selected:
> 
> - install a startup script.
> - install a simple configuration, using the ondemand process manager
>   that only starts children when the website is actually being used.

 Correct me if I'm wrong, but what is still missing is the configuration of the
webserver to actually use this, right? You still have to update your webserver
configuration to point to /var/run/php-fpm.sock to handle php scripts, right?

 If so, perhaps that could be explained in the help text in php/Config.in.


 It may also be good to explain in the commit log why you don't use the provided
default config file

> 
> Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>
> ---
>  package/php/php-fpm.conf | 14 ++++++++++++++
>  package/php/php.mk       | 15 +++++++++++++++
>  2 files changed, 29 insertions(+)
>  create mode 100644 package/php/php-fpm.conf
> 
> diff --git a/package/php/php-fpm.conf b/package/php/php-fpm.conf
> new file mode 100644
> index 0000000..2ffe595
> --- /dev/null
> +++ b/package/php/php-fpm.conf
> @@ -0,0 +1,14 @@
> +[www]
> +# Only start children when there are requests to be processed
> +pm = ondemand
> +# Terminate them again after there haven't been any for 2 minutes
> +pm.process_idle_timeout = 120s
> +# Maximum number of children processing PHP requests concurrently
> +pm.max_children = 32

 Isn't that a bit high? Typically embedded web servers will not have the power
to handle that many parallel requests efficiently.

> +
> +listen = /var/run/php-fpm.sock
> +listen.owner = www-data
> +listen.group = www-data
> +user = www-data
> +group = www-data
> +
> diff --git a/package/php/php.mk b/package/php/php.mk
> index e4331f2..6c42aba 100644
> --- a/package/php/php.mk
> +++ b/package/php/php.mk
> @@ -251,6 +251,21 @@ PHP_CONF_OPTS += \
>  PHP_DEPENDENCIES += jpeg libpng freetype
>  endif
>  
> +ifeq ($(BR2_PACKAGE_PHP_FPM),y)
> +define PHP_INSTALL_INIT_SYSV
> +	$(INSTALL) -D -m 0755 $(@D)/sapi/fpm/init.d.php-fpm \
> +		$(TARGET_DIR)/etc/init.d/S49php-fpm
> +endef

 There's also a php-fpm.service that you can install for systemd.


 Regards,
 Arnout

> +
> +define PHP_INSTALL_FPM_CONF
> +	$(INSTALL) -D -m 0644 package/php/php-fpm.conf \
> +		$(TARGET_DIR)/etc/php-fpm.conf
> +	rm -f $(TARGET_DIR)/etc/php-fpm.conf.default
> +endef
> +
> +PHP_POST_INSTALL_TARGET_HOOKS += PHP_INSTALL_FPM_CONF
> +endif
> +
>  define PHP_EXTENSIONS_FIXUP
>  	$(SED) "/prefix/ s:/usr:$(STAGING_DIR)/usr:" \
>  		$(STAGING_DIR)/usr/bin/phpize
>

Hi,

On 04/30/2015 10:27 PM, Arnout Vandecappelle wrote:
> On 04/30/15 21:36, Floris Bos wrote:
>> When PHP's FastCGI Process Manager SAPI is selected:
>>
>> - install a startup script.
>> - install a simple configuration, using the ondemand process manager
>>    that only starts children when the website is actually being used.
>   Correct me if I'm wrong, but what is still missing is the configuration of the
> webserver to actually use this, right? You still have to update your webserver
> configuration to point to /var/run/php-fpm.sock to handle php scripts, right?

Correct.
Although I am thinking about submitting a subsequent patch that adds an 
option to enable it in the webserver configuration.

I know that may go against the "tradition" that buildroot only does the 
compiling, and the user has to do his own configuration.
But things like enabling PHP in a webserver configuration is one of 
those annoying repetitive tasks that I really like to see handled by 
ticking a box.

>   If so, perhaps that could be explained in the help text in php/Config.in.
>
>
>   It may also be good to explain in the commit log why you don't use the provided
> default config file

The stock php-fpm config is good for illustrative purposes only.
It is unfit for multi-user systems, as it listens to a TCP socket with 
no authentication, allowing any local user that knows how to connect to 
127.0.0.1:9000 to elevate privileges and execute arbitrary code as the 
php user.
And it uses a pm that keeps a minimum number of PHP children resident at 
all times, making it not the best choice for embedded systems either.

>> Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>
>> ---
>>   package/php/php-fpm.conf | 14 ++++++++++++++
>>   package/php/php.mk       | 15 +++++++++++++++
>>   2 files changed, 29 insertions(+)
>>   create mode 100644 package/php/php-fpm.conf
>>
>> diff --git a/package/php/php-fpm.conf b/package/php/php-fpm.conf
>> new file mode 100644
>> index 0000000..2ffe595
>> --- /dev/null
>> +++ b/package/php/php-fpm.conf
>> @@ -0,0 +1,14 @@
>> +[www]
>> +# Only start children when there are requests to be processed
>> +pm = ondemand
>> +# Terminate them again after there haven't been any for 2 minutes
>> +pm.process_idle_timeout = 120s
>> +# Maximum number of children processing PHP requests concurrently
>> +pm.max_children = 32
>   Isn't that a bit high? Typically embedded web servers will not have the power
> to handle that many parallel requests efficiently.

Some of the web applications we use feature an AJAX web interface that 
uses a form of "long polling" to poll for certain events.
That essentially ties up a PHP process per logged-in user, so I like the 
number a bit higher than usual.
Doesn't consume much resources (at least not cpu), but does block for 29 
seconds if no events come in.
Also scripts that query data from external servers can block as well.

The number could be made a little lower, as not everybody does stuff 
like that.
But it doesn't really harm others either, as extra children are only 
started if all the existing ones are occupied. Not by default.


>
>> +
>> +listen = /var/run/php-fpm.sock
>> +listen.owner = www-data
>> +listen.group = www-data
>> +user = www-data
>> +group = www-data
>> +
>> diff --git a/package/php/php.mk b/package/php/php.mk
>> index e4331f2..6c42aba 100644
>> --- a/package/php/php.mk
>> +++ b/package/php/php.mk
>> @@ -251,6 +251,21 @@ PHP_CONF_OPTS += \
>>   PHP_DEPENDENCIES += jpeg libpng freetype
>>   endif
>>   
>> +ifeq ($(BR2_PACKAGE_PHP_FPM),y)
>> +define PHP_INSTALL_INIT_SYSV
>> +	$(INSTALL) -D -m 0755 $(@D)/sapi/fpm/init.d.php-fpm \
>> +		$(TARGET_DIR)/etc/init.d/S49php-fpm
>> +endef
>   There's also a php-fpm.service that you can install for systemd.

Yes, I noticed that before submitting the patch.
But I never had much luck with systemd on buildroot.

The .service file generated by the PHP build did not work as-is when I 
tried it this morning.

==
Konsole output [Unit]
Description=The PHP FastCGI Process Manager
After=syslog.target network.target

[Service]
Type=simple
PIDFile=/var/run/php-fpm.pid
ExecStart=${exec_prefix}/sbin/php-fpm --nodaemonize --fpm-config 
/etc/php-fpm.conf
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target
==

First problem was that systemd didn't like the ${exec_prefix}
And I am not sure if we have the syslog.target either.

But another thing I noticed was that my webserver and my other custom 
applications were not starting properly with systemd anymore.
Seems for some reason /tmp is not world-writable anymore.
So if your application does not run as root, and wants to create a file 
in say /var/log (that links to /tmp) it fails.
Haven't yet figured out what is up with that.


Yours sincerely,

Floris Bos

On 05/01/15 00:15, Floris Bos wrote:
> Hi,
> 
> On 04/30/2015 10:27 PM, Arnout Vandecappelle wrote:
>> On 04/30/15 21:36, Floris Bos wrote:
>>> When PHP's FastCGI Process Manager SAPI is selected:
>>>
>>> - install a startup script.
>>> - install a simple configuration, using the ondemand process manager
>>>   that only starts children when the website is actually being used.
>>  Correct me if I'm wrong, but what is still missing is the configuration of the
>> webserver to actually use this, right? You still have to update your webserver
>> configuration to point to /var/run/php-fpm.sock to handle php scripts, right?
> 
> Correct.
> Although I am thinking about submitting a subsequent patch that adds an option
> to enable it in the webserver configuration.
> 
> I know that may go against the "tradition" that buildroot only does the
> compiling, and the user has to do his own configuration.

 No, the tradition is that as much as possible it works out of the box, but in
the most simple way. As long as it is easy to customize in post-build and/or
overlay.

> But things like enabling PHP in a webserver configuration is one of those
> annoying repetitive tasks that I really like to see handled by ticking a box.

 I fully agree.

> 
>>  If so, perhaps that could be explained in the help text in php/Config.in.
>>
>>
>>  It may also be good to explain in the commit log why you don't use the provided
>> default config file
> 
> The stock php-fpm config is good for illustrative purposes only.
> It is unfit for multi-user systems, as it listens to a TCP socket with no
> authentication, allowing any local user that knows how to connect to
> 127.0.0.1:9000 to elevate privileges and execute arbitrary code as the php user.
> And it uses a pm that keeps a minimum number of PHP children resident at all
> times, making it not the best choice for embedded systems either.

 I fully agree, just thought it should be mentioned in the commit message.

> 
>>> Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>
>>> ---
>>>  package/php/php-fpm.conf | 14 ++++++++++++++
>>>  package/php/php.mk       | 15 +++++++++++++++
>>>  2 files changed, 29 insertions(+)
>>>  create mode 100644 package/php/php-fpm.conf
>>>
>>> diff --git a/package/php/php-fpm.conf b/package/php/php-fpm.conf
>>> new file mode 100644
>>> index 0000000..2ffe595
>>> --- /dev/null
>>> +++ b/package/php/php-fpm.conf
>>> @@ -0,0 +1,14 @@
>>> +[www]
>>> +# Only start children when there are requests to be processed
>>> +pm = ondemand
>>> +# Terminate them again after there haven't been any for 2 minutes
>>> +pm.process_idle_timeout = 120s
>>> +# Maximum number of children processing PHP requests concurrently
>>> +pm.max_children = 32
>>  Isn't that a bit high? Typically embedded web servers will not have the power
>> to handle that many parallel requests efficiently.
> 
> Some of the web applications we use feature an AJAX web interface that uses a
> form of "long polling" to poll for certain events.

 I thought long polling was exactly the opposite: close the connection and poll
again later. But I don't know about these things :-)

> That essentially ties up a PHP process per logged-in user, so I like the number
> a bit higher than usual.
> Doesn't consume much resources (at least not cpu), but does block for 29 seconds
> if no events come in.
> Also scripts that query data from external servers can block as well.
> 
> The number could be made a little lower, as not everybody does stuff like that.
> But it doesn't really harm others either, as extra children are only started if
> all the existing ones are occupied. Not by default.

 However, in a memory-constrained system (which still is one of buildroot's core
targets), it will trigger OOM when there actually are so many children.


> 
> 
>>
>>> +
>>> +listen = /var/run/php-fpm.sock
>>> +listen.owner = www-data
>>> +listen.group = www-data
>>> +user = www-data
>>> +group = www-data
>>> +
>>> diff --git a/package/php/php.mk b/package/php/php.mk
>>> index e4331f2..6c42aba 100644
>>> --- a/package/php/php.mk
>>> +++ b/package/php/php.mk
>>> @@ -251,6 +251,21 @@ PHP_CONF_OPTS += \
>>>  PHP_DEPENDENCIES += jpeg libpng freetype
>>>  endif
>>>  
>>> +ifeq ($(BR2_PACKAGE_PHP_FPM),y)
>>> +define PHP_INSTALL_INIT_SYSV
>>> +	$(INSTALL) -D -m 0755 $(@D)/sapi/fpm/init.d.php-fpm \
>>> +		$(TARGET_DIR)/etc/init.d/S49php-fpm
>>> +endef
>>  There's also a php-fpm.service that you can install for systemd.
> 
> Yes, I noticed that before submitting the patch.
> But I never had much luck with systemd on buildroot.
> 
> The .service file generated by the PHP build did not work as-is when I tried it
> this morning.
> 
> ==
> Konsole output [Unit]
> Description=The PHP FastCGI Process Manager
> After=syslog.target network.target
> 
> [Service]
> Type=simple
> PIDFile=/var/run/php-fpm.pid
> ExecStart=${exec_prefix}/sbin/php-fpm --nodaemonize --fpm-config /etc/php-fpm.conf
> ExecReload=/bin/kill -USR2 $MAINPID
> 
> [Install]
> WantedBy=multi-user.target
> ==
> 
> First problem was that systemd didn't like the ${exec_prefix}

 That should probably have been substituted away by the conf.in -> conf
conversion...

> And I am not sure if we have the syslog.target either.

 I believe journald provides that.

> 
> But another thing I noticed was that my webserver and my other custom
> applications were not starting properly with systemd anymore.
> Seems for some reason /tmp is not world-writable anymore.
> So if your application does not run as root, and wants to create a file in say
> /var/log (that links to /tmp) it fails.

 I guess whoever uses systemd will deal with that.


 Personally, I think it's worthwhile to install the service file even if it is
not working. If nobody actually uses it, there's no harm done. If someone does
use it, they're no worse off than if there was not service file. And they may
give us a patch that fixes the problem.

 Regards,
 Arnout

> Haven't yet figured out what is up with that.
> 
> 
> Yours sincerely,
> 
> Floris Bos
>

On 05/01/2015 01:20 AM, Arnout Vandecappelle wrote:
>>>> Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>
>>>> ---
>>>>   package/php/php-fpm.conf | 14 ++++++++++++++
>>>>   package/php/php.mk       | 15 +++++++++++++++
>>>>   2 files changed, 29 insertions(+)
>>>>   create mode 100644 package/php/php-fpm.conf
>>>>
>>>> diff --git a/package/php/php-fpm.conf b/package/php/php-fpm.conf
>>>> new file mode 100644
>>>> index 0000000..2ffe595
>>>> --- /dev/null
>>>> +++ b/package/php/php-fpm.conf
>>>> @@ -0,0 +1,14 @@
>>>> +[www]
>>>> +# Only start children when there are requests to be processed
>>>> +pm = ondemand
>>>> +# Terminate them again after there haven't been any for 2 minutes
>>>> +pm.process_idle_timeout = 120s
>>>> +# Maximum number of children processing PHP requests concurrently
>>>> +pm.max_children = 32
>>>   Isn't that a bit high? Typically embedded web servers will not have the power
>>> to handle that many parallel requests efficiently.
>> Some of the web applications we use feature an AJAX web interface that uses a
>> form of "long polling" to poll for certain events.
>   I thought long polling was exactly the opposite: close the connection and poll
> again later. But I don't know about these things :-)

Letting the client Javascript code poll at set intervals -e.g. every 
second- and the server returning a response immediately each time, is 
traditional polling.

With long polling you let the client wait LONG for the response instead.
The server (PHP code) delays sending the response until there is new 
data to report, or a timeout happens (29 seconds in our case) after 
which a dummy response is sent to keep the line open, and the client 
sends a new long poll request immediately.
Advantage is that the client receives the new data as soon as it is 
available, and does not have to have to wait until the next client poll 
interval, resulting in a smoother user experience.
Downside is that it esentially ties up at least one PHP process per 
logged-in webinterface user, just for the polling.

>
>> That essentially ties up a PHP process per logged-in user, so I like the number
>> a bit higher than usual.
>> Doesn't consume much resources (at least not cpu), but does block for 29 seconds
>> if no events come in.
>> Also scripts that query data from external servers can block as well.
>>
>> The number could be made a little lower, as not everybody does stuff like that.
>> But it doesn't really harm others either, as extra children are only started if
>> all the existing ones are occupied. Not by default.
>   However, in a memory-constrained system (which still is one of buildroot's core
> targets), it will trigger OOM when there actually are so many children.

So how many would you recommend as buildroot default?
2? 4? 8? 16?

>
>>>> +
>>>> +listen = /var/run/php-fpm.sock
>>>> +listen.owner = www-data
>>>> +listen.group = www-data
>>>> +user = www-data
>>>> +group = www-data
>>>> +
>>>> diff --git a/package/php/php.mk b/package/php/php.mk
>>>> index e4331f2..6c42aba 100644
>>>> --- a/package/php/php.mk
>>>> +++ b/package/php/php.mk
>>>> @@ -251,6 +251,21 @@ PHP_CONF_OPTS += \
>>>>   PHP_DEPENDENCIES += jpeg libpng freetype
>>>>   endif
>>>>   
>>>> +ifeq ($(BR2_PACKAGE_PHP_FPM),y)
>>>> +define PHP_INSTALL_INIT_SYSV
>>>> +	$(INSTALL) -D -m 0755 $(@D)/sapi/fpm/init.d.php-fpm \
>>>> +		$(TARGET_DIR)/etc/init.d/S49php-fpm
>>>> +endef
>>>   There's also a php-fpm.service that you can install for systemd.
>> Yes, I noticed that before submitting the patch.
>> But I never had much luck with systemd on buildroot.
>>
>> The .service file generated by the PHP build did not work as-is when I tried it
>> this morning.
>>
>> ==
>> Konsole output [Unit]
>> Description=The PHP FastCGI Process Manager
>> After=syslog.target network.target
>>
>> [Service]
>> Type=simple
>> PIDFile=/var/run/php-fpm.pid
>> ExecStart=${exec_prefix}/sbin/php-fpm --nodaemonize --fpm-config /etc/php-fpm.conf
>> ExecReload=/bin/kill -USR2 $MAINPID
>>
>> [Install]
>> WantedBy=multi-user.target
>> ==
>>
>> First problem was that systemd didn't like the ${exec_prefix}
>   That should probably have been substituted away by the conf.in -> conf
> conversion...

Also seems to happen when compiling PHP outside of buildroot.

==
max@lynx:/tmp/p/php-5.6.8/sapi/fpm$ cat php-fpm.service
[Unit]
Description=The PHP FastCGI Process Manager
After=syslog.target network.target

[Service]
Type=simple
PIDFile=${prefix}/var/run/php-fpm.pid
ExecStart=${exec_prefix}/sbin/php-fpm --nodaemonize --fpm-config 
${prefix}/etc/php-fpm.conf
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target

max@lynx:/tmp/p/php-5.6.8/sapi/fpm$ cat php-fpm.service.in
[Unit]
Description=The PHP FastCGI Process Manager
After=syslog.target network.target

[Service]
Type=@php_fpm_systemd@
PIDFile=@localstatedir@/run/php-fpm.pid
ExecStart=@sbindir@/php-fpm --nodaemonize --fpm-config 
@sysconfdir@/php-fpm.conf
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target

==

Is that a bug in PHP, or does systemd support variables in .service 
descriptions like this, and is the real problem perhaps that we are 
failing to set exec_prefix in some kind of global systemd environment file?


Yours sincerely,

Floris Bos

On 05/01/15 12:42, Floris Bos wrote:
> On 05/01/2015 01:20 AM, Arnout Vandecappelle wrote:
[snip]
>>>>> +# Maximum number of children processing PHP requests concurrently
>>>>> +pm.max_children = 32
>>>>   Isn't that a bit high? Typically embedded web servers will not have the power
>>>> to handle that many parallel requests efficiently.
>>> Some of the web applications we use feature an AJAX web interface that uses a
>>> form of "long polling" to poll for certain events.
>>   I thought long polling was exactly the opposite: close the connection and poll
>> again later. But I don't know about these things :-)
> 
> Letting the client Javascript code poll at set intervals -e.g. every second- and
> the server returning a response immediately each time, is traditional polling.
> 
> With long polling you let the client wait LONG for the response instead.
> The server (PHP code) delays sending the response until there is new data to
> report, or a timeout happens (29 seconds in our case) after which a dummy
> response is sent to keep the line open, and the client sends a new long poll
> request immediately.
> Advantage is that the client receives the new data as soon as it is available,
> and does not have to have to wait until the next client poll interval, resulting
> in a smoother user experience.
> Downside is that it esentially ties up at least one PHP process per logged-in
> webinterface user, just for the polling.

 OK, thanks for the explanation. Since even I had heard of the term long
polling, I guess this must be pretty common :-)

 That said, I think in many cases buildroot-based systems will be serving just
one or two clients in parallel (at least that was the case for all the projects
that I have worked on), so I still think it's better to set the default at 5
(which seems to be the default in the .conf file provided by the package as well).

[snip]
>>>>   There's also a php-fpm.service that you can install for systemd.
>>> Yes, I noticed that before submitting the patch.
>>> But I never had much luck with systemd on buildroot.
>>>
>>> The .service file generated by the PHP build did not work as-is when I tried it
>>> this morning.
>>>
>>> ==
>>> Konsole output [Unit]
>>> Description=The PHP FastCGI Process Manager
>>> After=syslog.target network.target
>>>
>>> [Service]
>>> Type=simple
>>> PIDFile=/var/run/php-fpm.pid
>>> ExecStart=${exec_prefix}/sbin/php-fpm --nodaemonize --fpm-config
>>> /etc/php-fpm.conf
>>> ExecReload=/bin/kill -USR2 $MAINPID
>>>
>>> [Install]
>>> WantedBy=multi-user.target
>>> ==
>>>
>>> First problem was that systemd didn't like the ${exec_prefix}
>>   That should probably have been substituted away by the conf.in -> conf
>> conversion...
> 
> Also seems to happen when compiling PHP outside of buildroot.

 Of course, what I meant was that there's something wrong with how the PHP build
system handles that file. Which is really weird, because it follows exactly the
same pattern as @localstatedir@ -> /var/run one line higher. Ah, now I see: we
provide --localstatedir explicitly to configure, while sbindir is at its default.

 In the init.d fragment they added exec_prefix=@exec_prefix@ to handle this.


> 
> ==
> max@lynx:/tmp/p/php-5.6.8/sapi/fpm$ cat php-fpm.service
> [Unit]
> Description=The PHP FastCGI Process Manager
> After=syslog.target network.target
> 
> [Service]
> Type=simple
> PIDFile=${prefix}/var/run/php-fpm.pid
> ExecStart=${exec_prefix}/sbin/php-fpm --nodaemonize --fpm-config
> ${prefix}/etc/php-fpm.conf
> ExecReload=/bin/kill -USR2 $MAINPID
> 
> [Install]
> WantedBy=multi-user.target
> 
> max@lynx:/tmp/p/php-5.6.8/sapi/fpm$ cat php-fpm.service.in
> [Unit]
> Description=The PHP FastCGI Process Manager
> After=syslog.target network.target
> 
> [Service]
> Type=@php_fpm_systemd@
> PIDFile=@localstatedir@/run/php-fpm.pid
> ExecStart=@sbindir@/php-fpm --nodaemonize --fpm-config @sysconfdir@/php-fpm.conf
> ExecReload=/bin/kill -USR2 $MAINPID
> 
> [Install]
> WantedBy=multi-user.target
> 
> ==
> 
> Is that a bug in PHP, or does systemd support variables in .service descriptions
> like this, and is the real problem perhaps that we are failing to set
> exec_prefix in some kind of global systemd environment file?

 I don't think systemd supports that. Setting it in a global environment file
wouldn't work (it works for us, but not for the generic systemd use case)
because every service could be installed into a different directory (like e.g.
on NixOS).

 So there are two options:

- Add execdir to the environment, i.e. add
Environment="exec_prefix=@exec_prefix@"

- Replace @sbindir@ with @EXPANDED_SBINDIR@.

 Both options are upstreamable. I think the second one is the best one. For
upstream it should be done for the other variables as well of course
(localstatedir, sysconfdir).


 BTW, you should probably split the patch into two patches, one to install the
startup script and another one to install the config file, since both are rather
independent of each other.

 Regards,
 Arnout



> 
> 
> Yours sincerely,
> 
> Floris Bos
> 
> 
>