---
title: Python agent logging
source: https://docs.newrelic.com/docs/apm/agents/python-agent/troubleshooting/python-agent-logging
---

## Problem

You can configure the Python agent to log output. This configuration allows the agent to track whether it is connecting to New Relic properly and if any errors occur. This information will be useful to New Relic Support if you experience problems.

## Solution

Detailed `debug` logging can help troubleshoot your standard Python agent installation.

> #### ⚠️ IMPORTANT
>
> Some partners use different procedures:
>
> -   [Debugging the Python agent with Heroku](https://docs.newrelic.com/docs/agents/python-agent/hosting-services/python-agent-and-heroku#debugging-the-python-agent)
> -   [Debugging the Python agent with OpenShift](https://docs.newrelic.com/docs/agents/python-agent/hosting-services/python-agent-openshift#debugging-the-python-agent)

To enable `debug` logging:

1.  Open your **newrelic.ini**, usually located within your app hierarchy.
2.  Uncomment `#log_file = /tmp/newrelic-python-agent.log`. Ensure you have write permissions to the log location, changing the path and file name if necessary. If there is no suitable file location, you can set `log_file` to `stderr`.
3.  Change `log_level` to `debug` (from `info`).

    > #### ⚠️ CAUTION
    >
    > Logging at `debug` can generate a lot of data very quickly. Monitor the size of your log file closely, changing `log_level` back to `info` as you finish troubleshooting.
4.  Save and close the file. Restart your app for the settings to take affect.
5.  Generate a few minutes of traffic to your app.
6.  If sending your log file to New Relic Support, attach your **newrelic.ini** to your support ticket, and tell Support your time zone.

> #### ⚠️ IMPORTANT
>
> When troubleshooting your New Relic Python agent, ensure it has been configured to generate debug level log files, and monitor the size of your log file closely. Using `log_level = debug` generates a lot of data very quickly. After reproducing your problem, return the log file to `log_level = info`.

### Log to a file [#logging-to-a-file]

The agent uses the Python logging module to output log messages. Output from the agent should therefore be factored into the overall logging strategy for your application.

If you are not using the logging module, the agent provides a simplified way to enable a log file for the Python agent. To use this, set the `log_file` and `log_level` options in the [agent configuration file](https://docs.newrelic.com/docs/agents/python-agent/installation-and-configuration/python-agent-configuration).

**Example:**

```ini
log_file = /tmp/newrelic-python-agent.log
log_level = info
```

The path provided for `log_file` should be writable to the user that your application runs as.

> #### 💡 TIP
>
> If using `Apache/mod_wsgi` the Apache user has restricted access to the filesystem. Create a special directory into which the log file can be placed which is writable to the Apache user. Using an absolute path, and not a relative path is recommended.

The log level used can be one of `error`, `warning`, `info` or `debug`. Under normal circumstances, use `info`. More verbose debug options are used for `debug`. Do not use these verbose debug options for an extended period of time. They can generate excessive output and the log file handler from the logging module is the standard file handler and does not perform any log file rotation.

### Log to standard error [#logging-to-standard-error]

For some hosting providers, it may not be possible to use a distinct log file for the agent. Configure the logging module to log to standard error output. This output is captured in the normal error log file for your hosting mechanism.

To do this within the agent configuration file:

```ini
log_file = stderr
log_level = info
```

The value `stdout` may also be used instead of `stderr`.

### Log all data (audit log) [#audit-log]

If you need to record and view information about [**all** data being transmitted](https://docs.newrelic.com/docs/agents/manage-apm-agents/configuration/log-audit-all-data-your-new-relic-agent-transmits) between the monitored process and the New Relic collector, you can enable [audit logging](https://docs.newrelic.com/docs/apm/agents/python-agent/configuration/python-agent-configuration/#audit-log-file) for short periods of time. This is useful, for example, with debugging or auditing, when you need detailed information about what exactly is being transmitted.

### Troubleshoot log module conflicts [#logging-module-conflicts]

If no logging is captured, there may be a conflict with the way in which the Python logging module is being initialized and/or configured. Problems may arise using the following functions:

**`logging.config.fileConfig()` function**

One problem which can arise is when the monitored application is using the `logging.config.fileConfig()` function to initialize the Python logging module. If the Python logging module is initialized with this call, this will by default disable the agent logging if the agent had already been initialized prior to the call.

If using Python 2.6 or higher and the call to `logging.config.fileConfig()` is under your control, one option is to modify the call to pass the argument `disable_existing_loggers` with the value of `False`. This will prevent the existing loggers being disabled and as such the agent loggers will be left alone.

````py
logging.config.fileConfig('logging.cfg', disable_existing_loggers=False)
```

If you are unable to modify the call, or it would not be appropriate to do so, the alternative is to reinstate the logger installed by the agent. To do this, a logger section should be added to the logging configuration file being passed to `logging.config.fileConfig()`.

```ini
[loggers]
keys = root,newrelic

[logger_newrelic]
qualname = newrelic
level = INFO
handlers =
```

In this case, with the `handlers` setting being left unset, how messages will be logged will be dictated by the handler associated with the root logger. You can alternatively refer directly to the appropriate handler which should be used to log the messages.

A similar issue as above can also arise when the monitored application is using `logging.config.dictConfig()`.

````

**`disable_existing_loggers` function argument**

When using a dictionary to configure the logging module, there is no function argument `disable_existing_loggers`. Instead, specify it as a key within the dictionary.

````py
LOGGING = {
  'disable_existing_loggers': False,
  ...
}
```

Similar to before, to avoid inherited existing loggers, an explicit logger definition should be added to the dictionary used to configure the logging module.

```py
LOGGING = {
  ...,

  'loggers': {
    'newrelic': {
      'level': 'INFO',
    },
    ...
  }
}
```

For more information, see the [Python logging module](http://docs.python.org/library/logging.config.html) documentation.

````

Refer to the documentation of any web framework or application being used. It may provide a specialized mechanism for configuring the Python logging module. For example, when using Django, the dictionary approach to setting up the Python logging module is supported automatically, with the definitions being set in the `LOGGING` attribute within the Django settings module.

For more information, see the [Django logging configuration](https://docs.djangoproject.com/en/dev/topics/logging/#configuring-logging) documentation.

### Rotate agent log file [#rotation-of-agent-log-file]

When using the `log_file` option in the agent configuration, the standard file handler from the logging module is used. This does not do any log file rotation. Log file rotation is not done automatically as we will not know what sort of rotating log file handler you may want to use, plus the standard rotating log file handlers supplied with Python are not necessarily safe for a multi process configuration. As such, it may be necessary to download and use one of the third party rotating log file handlers or use a dedicated logging system.

If your application runs in a single process you can safely use either of the `RotatingFileHandler` or `TimedRotatingFileHandler` handlers supplied with the logging module. To use this only for the output from the Python agent, include at the start of your WSGI script file or module, but before the import of the `newrelic` module, the following:

```py
_LOG_FORMAT = (
    "%(asctime)s (%(process)d/%(threadName)s)" " %(name)s %(levelname)s - %(message)s"
)

_logger = logging.getLogger("newrelic")
_handler = logging.handlers.TimedRotatingFileHandler(
    "agent.log", when="midnight", backupCount=7
)
_formatter = logging.Formatter(_LOG_FORMAT)
_handler.setFormatter(_formatter)
_logger.addHandler(_handler)
_logger.setLevel(logging.INFO)
```

This code accesses the root `newrelic` logger instance, attaches the rotating log file handler to it, and sets the log level for messages to be sent to this log file. We also show how to set up the log message format, but that is optional.

If the logging module as a whole is initialized, any log output will propagate up and be logged by any handler associated with the root logger, including possibly to standard error. To avoid duplicates, configure the log handler for the root logger.

The above code could also be simplified by using the `logging.fileConfig()` function and a configuration file. For further details of using a configuration file see Python logging module documentation on [Logging configuration](http://docs.python.org/howto/logging.html#configuring-logging).

### Rotate log in multi-process configuration [#multi-process-safe-log-rotation]

The rotating log file handlers provided in the standard logging module are not entirely safe to use in a multi-process server configuration. Problems that can occur are intermingling of messages from multiple processes and attempts by multiple processes to perform log file rotation at the same time.

For a more robust log file rotation mechanism, use a third party log handler in conjunction with the Python logging module.

One such implementation available on PyPi is:

-   [Concurrent Log Handler](http://pypi.python.org/pypi/ConcurrentLogHandler/)

For a more complex solution you may also consider a logging service such as:

-   [Python Logging Server](http://code.google.com/p/python-loggingserver/)

Also consult the list of [other handlers](http://docs.python.org/library/logging.handlers.html#module-logging.handlers) provided as standard in the Python logging module as those for sending to a socket or posting to a HTTP URL may also be reasonable solutions in some circumstances.