Skip to content

Logging

It's a best practice to capture Policy Service pod logs. The Put and Delete APIs along with the corresponding (AUDIT messages added in M19) probably should be maintained. In M23 an additional audit log was added.

It's also a best practice to have logging turned on for updates for your backend bundle service. For example, if using AWS S3 as backend bundle service, have CloudTrail turned on with write events to the bucket or bundle.tar.gz, and .log objects.

Logging is currently supported two ways, via pod logs and an external audit log.

Pod Logs

Pod logs support the normal syslog severitys.

The default log level threshold is INFO. You can set this via the environment variable LOG_LEVEL. Logging messages which are less severe than level will be ignored; logging messages which have severity level or higher will be emitted by whichever handler or handlers service this logger, unless a handler’s level has been set to a higher severity level than level.

Additionally you can set a log level threshold for translate messages via LOG_LEVEL_TRANSLATE. By default it will use the same level as LOG_LEVEL. This might be useful for debugging specific translate operations.

It is never recommend to use DEBUG in a production environment as it may impact performance.

External Audit Logs

In M23 an external audit log was added to Policy Service. This is on by default but can be disabled by setting environment variable ENABLE_EXTERNAL_AUDIT_LOG to false. For every audit change event (Create, Update or Delete request) a new log file with additional details will be created at the root of the POLICY_BUCKET.

Log files will be named audit-Data Partition-YYYYMMDD-HHMMSS.log An example file name might be audit-osdu-20240301-112233.log It is recommended to have life-cycle like rules to delete this log files when they are no longer needed, for example after 90 days.

Each entry in the file will contain a line with the following details: YYYY-MM-DD HH:MM:SS,MS LEVEL [filename:lineno - funcName()] AUDIT: audit details

Audit details may include (if available):

  • request details. For example if it's a delete request, the action (delete policy) along with the policy name. For put requests in may also include the SHA256 (Secure Hash Algorithm) of the policy (if available).
  • correlation-id (from request headers, or assigned if none given). This correlation id will be used by policy service with calls to entitlement, legal, etc.
  • user_id (returned from entitlement service)
  • xuserid (from request headers)
  • requestid (from x-request-id in request headers)
  • Zipkin b3-propagation details: x-b3-traceid, x-b3-spanid, x-b3-parentspanid. See b3-propagation for more details.
  • forward (x-forwarded-for)[https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For]

Example log entry:

2024-02-09 20:49:02,692 INFO    [policy_update_api.py:387 - create_or_update_partition_policy()] AUDIT: put policy_id: search_issue_103_tc1.rego id: osdu/partition/opendes/search_issue_103_tc1.rego correlation-id: 065c68fb-ced3-7774-8000-98061af3668a user: 4114a503-7d3e-4ea3-b15d-fc8aa0056e8c xuserid: 4114a503-7d3e-4ea3-b15d-fc8aa0056e8c sha256: d0cbb75baa3e06518f215854c077318e61f56f06 requestid: e06df2e8-3f8f-4fe8-8e1b-b07bfb2c1c7b traceid: b76f8be2e7a667c3f038a470553df5cd spanid: 22bb98c8d2324c5b parentspanid: f038a470553df5cd forward: 170.39.106.43:36490,10.10.4.231