Bundles

Warning

With bundles enabled (the default), it is important that you validate the rego contained in the bootstrap file and when updating policies via policy service APIs.

If there are syntax errors in the bundle, OPA will refuse to load them into memory which could break the policy service. In M21 rego validation was introduced for PUT API requests, which is on by default but can be optionally disabled (see `ENABLE_VERIFY_POLICY`` for more information).

Types of bundles and bundle bootstrapping

Two types on bundles are bootstrapped: instance and partition bundles. Instance bundle cannot be changed through policy service but only by directly changing the instance bundle file in the bundle server. Partition bundle can be changed via policy service.

All bundles (instance and all partition bundles) need to be bootstrapped in bundle server.

Bundles

Naming conventions

Naming convention for these bundles:

bundle.tar.gz for instance the bundle and
bundle-data-partition-id.tar.gz for each data partition.

Instance bundles

Instance bundle contains .manifest with following contents:

{
    "roots": ["osdu/instance"]
}

Instance policies should be osdu/instance/xyz.rego. No extra / in the path. Strange results may occur if addition paths are added.

Partition Bundles

Partition bundle contains .manifest with following contents:

{
    "roots": ["osdu/partiton/data_partition_id"]
}

Partition policies should be osdu/partition//xyz.rego. No extra / in the path. Strange results may occur if addition paths are added.

To define a policy within a data partition, one must include a namespace that is specific for the given data partition. For example, dataauthz policy for data partition data_partition_id can look like this:

package osdu.partition["dp1-100"].dataauthz

import data.osdu.instance.dataauthz

records := data.osdu.instance.dataauthz.records

Examples of the above can be found in tests template directory.

Rate limiting

Warning

Policy Service does not implement a retry mechanism when writing to the bundle service. In environments where you expect a lot of policy additions, updates and deletes (i.e. test environments) special care should be taken when using rate limits to your bundle service backend as these put/deletes will fail, which could cause your testing to fail with "Error when talking to bundle service" in detail of API response along with a HTTP_503_SERVICE_UNAVAILABLE status code.

Adding a new partition to OSDU

Warning

It is currently required to have backend CSP access and complete these two steps prior to adding a new partition or your new partition in OSDU will likely be unusable.

Two things must be done when adding a new partition to OSDU before (or after) running the add partition API in OSDU.

Create a bundle for partition

Creating a bundle can be done two ways:

Calling the /boostrap Policy API
Update the backend storage directly. Put a bundle in bundle server (S3 or other storage location) for the partition. This bundle-data-partition-id.tar.gz should contain the following at a minimum:
- .manifest,
- dataauthz, and
- search

Update OPA

Update the OPA configuration so that it knows it should read in the new bundle for the partition that will be created. To do this you need to provide OPA a new config.yaml. However this is CSP independent today, but in general can be done by updating the OPA config map. It is recommended to update your existing config map running in your kubernetes env. Once the config map in Kubernetes has been updated the OPA pods/service should be restarted to force the read-in of this configuration. Review OPA pod logs for errors after updating.

Testing OPA configuration locally

You can test the opa config using the OPA CLI. The configuration file can be in JSON or YAML format. Please note some documentation may refer to it as config.yaml or init.yaml - they mean the same thing. Not to be confused with kubenetes config map yaml - This is just the data config section of config map.

opa run -s -c config.yaml

Policy API for retrieving configuration

The Policy /diag/config API (which is likely disabled in production environments), allows you to retrieve the configuration detail of OPA.

Automation for creating the deployed config map

Original template automation for config map:

M14-M17 AWS Config map helm charts - Please note in M18 AWS switched to Terraform.
Azure Config map
GC

Review current config map

If you have backend CSP access you can review the OPA config map:

Example from AWS:

kubectl -n osdu-services describe configmap opa-config

kubectl -n osdu-services get configmaps opa-config -o yaml > configmap.yaml

Updating config map

An example opa_config:

services:
  s3:
    url: https://r3m16-515517605230-us-east-1-policy.s3.amazonaws.com
    credentials:
      s3_signing:
        web_identity_credentials:
          region: us-east-1
bundles:
  osdu/instance:
    service: s3
    resource: bundle.tar.gz
    polling:
      min_delay_seconds: 6
      max_delay_seconds: 12
  osdu/partition/osdu:
    service: s3
    resource: bundle-osdu.tar.gz
    polling:
      min_delay_seconds: 6
      max_delay_seconds: 12

You will need add the following section:

  osdu/partition/<data partition>:
    service: <bundle service>
    resource: bundle-<data partition>.tar.gz
    polling:
      min_delay_seconds: 6
      max_delay_seconds: 12

Replace

bundle service can be: s3, gcs, azure_storage_account, etc.

data partition is the new data partition you are adding.

Warning

All values for min_delay_seconds and max_delay_seconds are supported, but you might need to adjust BUNDLE_PAUSE if you are running automated integration tests in this new data partition.

Applying Update

Warning

You may not want to do this if your environment was created with cloudformation, helm or terraform. Use those methods to avoid conflicts.

Interactive update of config map via kubectl (not recommended)

This can be tricky because of the spaces and newline \n in the data config section:

data:
  config: "services: \n  s3:\n    url: https://r3m16-515517605230-us-east-1-policy.s3.amazonaws.com\n
    \   credentials:\n      s3_signing:\n        web_identity_credentials:\n          region:
    us-east-1\nbundles: \n  osdu/instance:\n    service: s3\n    resource: bundle.tar.gz\n
    \   polling:\n      min_delay_seconds: 6\n      max_delay_seconds: 12\n  osdu/partition/osdu:\n
    \   service: s3\n    resource: bundle-osdu.tar.gz\n    polling:\n      min_delay_seconds:
    6\n      max_delay_seconds: 12"

Interactive update example:

kubectl -n osdu-services edit configmap opa-config

Apply Modifications from file using kubectl

kubectl apply -f configmap.yaml

Check for errors

Example, your pod name and namespace may be different

kubectl -n osdu-services logs  opa-agent-5ddf6fc476-lwp6p

Helm

If your environment is managed by helm. Get RELEASE_NAME.

helm list -A

Values for helm

helm get values core -n osdu-services

If there isn't a section like this than you are currently only supporting the default partition osdu:

opa-agent:
  partitions:
  - osdu
  - other partitions

Add the above to file opa-values.yaml

Then update using helm (dry-run first):

helm upgrade core core/core --reuse-values --version 0.19.1 -f opa-values.yaml --dry-run

If no errors, for real:

helm upgrade core core/core --reuse-values --version 0.19.1 -f opa-values.yaml

Terraform

OSDU M18 on AWS is expected to use Terraform for managing helm which controls the config map for OPA. When this is released these instructions will be updated.