IOError: permission denied handling USR1 signal after switching to unprivileged user

[parasyte]

Since I hit the file descriptor leak in #1012, I switched to using USR1 signal in logrotate instead of HUP. I got it working until I added user and group config variables to switch to an unprivileged user in the worker.

When gunicorn master starts, it creates the access and error log files owned by root, and the children inherit these FDs and continue writing to the privileged FDs even after switching to an unprivileged user. (Weird, but AFAICT this is expected behavior.)

The problem comes when the USR1 signal is sent; the unprivileged workers attempt to reopen a file that is owned by root!

I'm using gevent, which explains this particular traceback showing up in cffi...

From cffi callback <bound method loop._check_callback of <loop at 0x1b62170 epoll default
pending=0 ref=9 fileno=10>>:
Traceback (most recent call last):
  File "/env/site-packages/gevent/corecffi.py", line 597, in _check_callback
    if self.ate_keyboard_interrupt:
  File "wsgi.py", line 484, in _handler
    if callable(last) and last(signum, frame) == False:
  File "/env/site-packages/gunicorn/workers/base.py", line 144, in handle_usr1
    self.log.reopen_files()
  File "/env/site-packages/gunicorn/glogging.py", line 301, in reopen_files
    handler.mode)
IOError: [Errno 13] Permission denied: '/path/to/error.log'

[parasyte]

For anyone who happens upon this ticket in the future: I've gone back to using setuidgid from daemontools to launch my service as the correct unprivileged user.

[tilgovi]

Interesting. Not sure what to do here. @benoitc any thoughts?

[benoitc]

I didn't test it yet. I looked at the code and it's supposed to reopen using the same mode:
https://github.com/benoitc/gunicorn/blob/master/gunicorn/glogging.py#L301-L302

Setting the correct umask should do the trick.

[parasyte]

umask will just hide the problem. I suggest using os.chown on the log files prior to fork.

[benoitc]

the arbiter also need to access to the logs... this is why the umask and
correct perms are needed there. btw are you doing USR1 on the workers or
the arbiter?
On Tue, 22 Sep 2015 at 07:29, Jay Oster [email protected] wrote:

umask will just hide the problem. I suggest using os.chown on the log
files prior to fork.

—
Reply to this email directly or view it on GitHub
#1116 (comment).

[parasyte]

USR1 is sent just to the master process (arbiter?)

I'm running gunicorn as root, so it will always have write access to the log files, regardless of the owner. As AFAIK, only root is allowed to use os.setuid anyway...

[benoitc]

@parasyte but if you spawn workers under a different user they also need to have write access to the logs since they are writing directly to it. What is your exact command line right now?

The os.chown is a good idea. I was thinking that the logging module was doing it when the log was created. I will have a look. Not that the daemontools also rely on a umask, but less restrictive.

Note: Eventually that can be changed if we spawn a specific process acting as a proxy to write logs. this maybe something to investigate as a separate ticket.

[parasyte]

daemontools works because it drops root privileges before gunicorn is started. So gunicorn is running under the unprivileged user, and creates the log file with the correct owner.

The bug described in this ticket occurs when gunicorn is started as root (creates the log files with root as owner) then drops privileges before the fork. It can be replicated with something like:

# wsgi.py
from wsgiref.simple_server import make_server, demo_app

if __name__ == '__main__':
    httpd = make_server('', 8000, demo_app)
    print "Serving HTTP on port 8000..."

    httpd.serve_forever()

Run it as root and drop privileges:

$ sudo gunicorn --user nobody --group nogroup --access-logfile /var/log/gunicorn.log --pid /var/run/gunicorn.pid wsgi:demo_app

Now send USR1 signal to the master process:

$ sudo kill -USR1 $(cat /var/run/gunicorn.pid)

See IOError raised in terminal.

[benoitc]

fixed. Thanks!