Skip to content

Release Notes for Policy Service

M24 Release 0.27

Policy Service v0.27.0 2024/07

M24 Minor Changes

M24 Bug Fixes

M24 Features

M23 Release 0.26

Policy Service v0.26.0 2024/04

M23 Minor Changes

  • Version numbers have changed to conform with OSDU 126. API reports version 1.0.0
  • Updated the default DataAuthz and Search policies to treat 'users.data.root' as super user. Please see the request in Storage service 220.

M23 Bug Fixes

  • Upgraded FastAPI to 0.109.2 for security fixes CVE-2024-24762
  • Upgraded usage of SHA1 to SHA256. This includes detection of changes to policies, response headers and response json if SHA-1 was previously included. 124 and 125. This SHA-256 will also show up in pod audit logs and external audit logs when relevant.
  • Updated OPA recommended versions for CVE-2024-24783, CVE-2023-6246, and CVE-2023-6779.
  • Added x-user-id header to default data authorization rego policies to fix issue where OPA's requests to entitlements and legal service were not authorized. Issue 129.

M23 Features

  • Separate Audit Log 113. See logging for more details.

M22 Release 0.25

Policy Service v0.25.0 2024/01

M22 Bug Fixes

  • Policy Service not all return status codes that are specified in OpenAPI.json 118
  • Policy /tenant API bug 119
  • Bugfix for Policy validation for non-ascii characters/utf-8 122. Added integration pipeline test for this as well.
  • Corrected Documentation issues

M22 Minor Changes

  • Updated Documentation 123
  • Default policy change for updates, to no longer check if legal tag is valid 678

M22 Features

  • New optional group support for /boostrap and /tenant APIs. 121. By default it will use policy.service.admin but this can be changed with environment variable OPS_PERMISSION. Updated OpenAPI/swagger to show required permissions for all APIs as configured and related documentation.

M21 Release 0.24

Policy Service v0.24.0 2023/10

M21 Breaking Changes

  • Put /policies/osdu/partition/{data_partition}/{policy_id} will now return 202 for accepted updates to policies, that is updated and new policies. If no change is detected it will return a 200. Additionally the return json has changed slightly (message updates, new validated and new SHA1 in response).

M21 Bug Fixes

  • X-Debug-Result response header bugfix for use with ENABLE_DEV_DIAGNOSTICS
  • Policy Service should handle entitlement service issues gracefully 112
  • Bugfix for LOCAL mode (developer use) on /config API

M21 Issues

  • Policy validation will fail if it contains non-ascii characters/utf-8 even if the rego is valid. This has been fixed in M22. See 122

M21 Minor Changes

  • Log message updates
  • Improved AUDIT logging
  • It is now recommended to use OPA version 0.56.0

M21 Features

  • New API /validate/{policy_id} with template support to check rego syntax and validate package naming within the rego.
  • Rego policies are now validated by default when uploading new policies (put /policies/osdu/partition/{data_partition}/{policy_id}). Validation can be turned off by setting Environment variable ENABLE_VERIFY_POLICY to False.
  • For put /validate/{policy_id}, put /policies/osdu/partition/{data_partition}/{policy_id} and get /policies/osdu/partition/{data_partition}/{policy_id} there is a new response header X-SHA-1 which is the SHA-1 of the respective policy. This SHA1 also shows up in pod AUDIT logs with changes.
  • Put /policies/osdu/partition/{data_partition}/{policy_id} now includes this SHA-1 in json response.
  • When adding/updating policies Put /policies/osdu/partition/{data_partition}/{policy_id}, policy service will now detect if this is a new policy vs update and will update the message field in the json return accordingly. If new "added to ...", if updated "updated in...". If there is no change, it will respond with a "already in..." message field. The response codes (202) will be given for new or updated policies. A 200 will be returned if there was no change detected.
  • Envoy distributed tracing support: Request ID (x-request-id), external client (x-client-trace-id), Zipkin B3 Propagation (x-b3-traceid, x-b3-span-id), OTEL/Lightstep/ServiceNow tracers (x-ot-span-context), Datadog tracer (x-datdog-trace-id), SkyWalking tracer (sw8), and AWS X-Ray tracer (x-amzn-trace-id). These headers will also become available in response headers if they are part of the request headers to assist with distributed tracing. Distributed tracing allows developers to obtain visualizations of call flows in large service oriented architectures. Envoy support can disabled by envionment variable ENVOY.
  • x-envoy-upstream-service-time may now be included in response headers if supported by the environment. This response contains the time in milliseconds spent by the upstream host processing the request (policy service) and the network latency between Envoy and upstream host (policy service). This along with X-Process-Time added in M20 help with performance timings.

M20 Release 0.23

Policy Service v0.23.0 2023/09

M20 Breaking Changes

M20 Bug Fixes

  • Performance bug fix memory leak - this bug is very significant in M18 and M19. Upgrading to M20 is highly recommended.
  • Bugfix for development MOCK_ENTITLEMENT
  • Bugfix for /config API not reporting all configuration details.
  • Bugfix for Translate API deny rule only MR
  • Bugfix conf USER_PERMISSION update

M20 Minor Changes

  • Cache settings can now be tweaked by environment variables to the policy service (POD). See Testing
  • Added Audit information for /bootstrap API
  • Logging updates
  • Updated error responses to APIs, to make error messages more clear.
  • Added more configuration details in /config API response, including uptime of pod, pod name.
  • ENV details can not be masked/unmasked in /config API response via env variable MASK_CONFIG_ENV.
  • New support for additional ENV variables including NAMESPACE to override namespace where OPA is located and OPA_CONFIG_MAP.
  • Upgraded FastAPI to release 0.101.1 and Pydantic 2.1.1 for improved correctness in corner cases, safer types, and better performance and less energy consumption. This upgrade also brought OpenAPI 3.1.0, this uses JSON Schema 2020-12 and required Swagger UI bump to version 5.x.x (to support OpenAPI 3.1.0)
  • Other library changes - to support yaml, kubernetes and gathering pod details.
  • Updated integration tests
  • Updated Azure build
  • Updated documentation

M20 Features

  • /tenant APIs for retrieving and updating data partition configuration details for OPA for ADR. In future these APIs will support the addition of new partitions. These require new permissions in kubernetes, see OPA documentation role and service token for more details.

    • Get /tenant - retrieve configuration details for partition.
    • Put /tenant - update configuration details for partition. You can only update existing details, new configuration details can not be added in M20. Expect more improvements in M21 here.
    • Delete /tenant - not implemented yet, but a stub exists - to be added in M21.

  • Added option in Translate API to construct elastic search subquery in search rego policy.

  • Added new response header X-Process-Time to all API call responses from policy service. This new headers contains the time in seconds that it took to process the request and generate a response in policy service. This new response header can be extremely useful for helping diagnosing performance issues and removing other infrastructure/internet delays and variance. For more details on API Response headers see API Documentation.
  • New diagnostic API /diag/tracker for checking object changes in memory over time (memory leak detection), this like all other /diag APIs are off by default and are not intented to be used in production environments.

M20 Issues

  • Policy service does not check if policy rego is valid before saving it to the bundle. It is therefore possible to "break" OPA and thereby Policy Service by writing a corrupt policy, which in turn breaks Search and Storage. It is recommended to check rego validity before uploading to policy service. Please note OPA CLI and AdminCLI supports this with check command.
  • Policy Service should handle entitlement service issues gracefully 112

M19 Release 0.22

Policy Service v0.22.0 2023/07

M19 Breaking Changes

None

M19 Bug Fixes

  • Multiple performance fixes
  • Bugfix for /eval and /compile APIs based upon spec-based API tests

M19 Minor Changes

  • Added Correlation-ID (X-Correlation-ID) in response headers for most API calls
  • Added more details to /config API return, including memory, cpu usage, and k8s config map details
  • Logging updates
  • Documentation updates
  • Updated library dependencies
  • Updated integration tests and load tests
  • Updated Azure build - for improved performance
  • Updated GC deployment configuration details for pipeline and preship
  • Translate API input validation update and better error handling
  • Configuration details can also be provided by .env files - useful for developers, on-premise and other testing
  • Baremetal support (Anthos) updates
  • Added CLOUD_PROVIDER check to /backup API for LOCAL
  • Added file payload check to compile API and better error handling
  • Added file validate payload to evaluation API. If validation fails HTTP_422_UNPROCESSABLE_ENTITY will be returned.

M19 Features

  • Multiple Performance fixes
  • Added audit information for when policies are updated or deleted.

M19 Issues

  • Memory leak - In M18 and M19 there is a memory leak from a library. This was fixed in M20. There is no fix for M19 at this time. A workaround is to increase number of pods and/or restart pods via: kubectl rollout restart deployment <policy_deployment_name> -n <namespace>
  • Policy service does not check if policy rego is valid before saving it to the bundle. It is therefore possible to "break" OPA and thereby Policy Service by writing a corrupt policy, which in turn breaks Search and Storage. It is recommended to check rego validity before uploading to policy service. Please note OPA CLI and AdminCLI supports this with check command.
  • Policy Service should handle entitlement service issues gracefully 112

M18 Release 0.21

Policy Service v0.1.4 2023/05

M18 Breaking Changes

None

M18 Minor Changes

  • Upgrade fastapi to 0.95.2 and Starlette version to >=0.27.0 for a security release. PR #9541. Details on Starlette's security advisory.
  • Updated /config to attempt to retrieve opa configmap details (if kubernetes security token file is present)
  • Vulnerability fix Cross-realm object access in Webpack 5 for built-in Policy AdminUI POC which is disabled by default
  • Logging changes
  • Support for comments in test templates used in integration testing
  • OPA recommended version is 0.54 to address numerous bugs, security and performance issues.
  • Allow caching to be disabled via environment variable DISABLE_OPA_CACHE

M18 Features

  • Translate API pre-processor (enabled by default, but can be disabled by env variable ENABLE_TRANSLATE_PREPROCESSOR). This allows Policy translate (which is used by search) to support more complex queries and deny policies
  • Translate API change - groups are now optional in Input of TranslateItem. See updated openapi.yaml. This should make this API easier to use and increase performance.
  • Backup API to partly address AWS and Azure
  • Bootstrap API to for partly addressing adding new partitions and
  • Policy Service now has the ability to verify policies before saving them to bundle for Opa Sync issue
  • Config API show policy service configuration, formerly as /diag/config and used only by developers. It was previously off by default. With even more configuration options it's now necessary to expose some of those details.

M18 Issues

  • Policy service does not check if policy rego is valid before saving it to the bundle. It is therefore possible to "break" OPA and thereby Policy Service by writing a corrupt policy, which in turn breaks Search and Storage. It is recommended to check rego validity before uploading to policy service. Please note OPA CLI and AdminCLI supports this with check command.
  • Performance issue - In M18 a performance bug was introduced. This was fixed in M19.
  • Memory leak - In M18 and M19 there is a memory leak from a library. This was fixed in M20. There is no fix for M18 at this time. A workaround is to increase number of pods and/or restart pods via: kubectl rollout restart deployment <policy_deployment_name> -n <namespace>
  • Policy Service should handle entitlement service issues gracefully 112

M17 Release 0.20

Policy Service v0.1.3 2023/03

M17 Breaking Changes

None

M17 Minor Changes

  • Logging changes
  • Updated documentation - including new sizing recommendations and livenessProbe
  • Translate update for search
  • Test updates
  • /health API no longer checks status of OPA
  • New /ready API checks status of OPA
  • GC changes, (GCP to GC)

M17 Issues

  • /translate API - is not able to handle really complex queries and deny policies
  • Policy service does not check if policy rego is valid before saving it to the bundle. It is therefore possible to "break" OPA and thereby Policy Service by writing a corrupt policy, which in turn breaks Search and Storage. It is recommended to check rego validity before uploading to policy service. Please note OPA CLI and AdminCLI supports this with check command.

M16 Release 0.19

Policy Service v0.1.2 2023/02

M16 Breaking Changes

None

M16 Minor Changes

  • Added ability to mock entitlement service result MOCK_ENTITLEMENT - useful for development/local testing.
  • Improved documentation
  • Removed authentication requirement from /info API endpoint

M16 Bugs

  • Security DOS fix - MultipartParser DOS with too many fields or files. Updated packages including FastAPI, Starlette, etc.

M16 Features

  • Caching - Enabled caching of results of OPA by default. This can be disabled and is configurable. Caching increases memory needs of Policy Service pods. Additionally timing of OPA responses are now logged. See Install Notes for updated memory request recommendations.

M16 Issues

  • /translate API - is not able to handle really complex queries and deny policies
  • Policy service does not check if policy rego is valid before saving it to the bundle. It is therefore possible to "break" OPA and thereby Policy Service by writing a corrupt policy, which in breaks Search and Storage.

M15 Release 0.18

Policy Service v0.1.1 2022/12

In M15 Policy Service was turned ON by default for AWS and Azure. See M15 release notes for more

M15 Breaking Changes

  • /compile API - used by policy developers, no longer requires developer mode (ENABLE_DEV_DIAGNOSTICS)

M15 Minor Changes

  • ENABLE_DEV_DIAGNOSTICS no longer enables diagnostic dumps, ENABLE_DEV_DIAGNOSTICS_DUMPS added for development/testing purposes.
  • Logging improvements and updates
  • Requirements update
  • Updated integration tests and testing options
  • Various bootstrapping and pipeline updates
  • Loadtesting improvements

M15 Bugs

  • Storage/bundle bugfix
  • Opa bugfix

M15 Issues

M15 Features

  • Correlation-ID support - Correlation-id header accepted in all inbound API calls and outbound API calls to other OSDU services. If no correlation-id is available in the headers, one will be created to track calls. This created correlation-id is based upon UUID version 7, generating time-sorted UUIDs with 200ns time resolution and 48 bits of randomness.
  • /translate - return status and error message changes to support search better
  • Anthos/MinIO support for bundle/backend storage
  • Autoscaling of OPA and Policy Service in AWS

M15 Docs

  • In M15, at the advice of OSDU community, what was the Policy AdminCLI was split off into it's own repo and renamed.
  • Updated documentation, including updated OpenAPI documentation.

M14 - Initial Release

M14 Release Notes