Versions Compared
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Configuring logging is covered Zone Management: Logging page.
Former user (Deleted) We might want to break this out into two pagesLogging
Table of Contents |
---|
YOUnite Uses Elastic
YOUnite logs directly to Elastic via the Elastic RESTtful API.All activity through the MDM RESTful API is logged synchronouslyuses the Open Source Elastic Stack (https://www.elastic.co/products) to log (Logstash), search (Elasticsearch), analyze and visualize (Kibana) all API requests and message bus traffic (most notably all data events).
YOUnite logs directly to Elastic via Elastic's RESTtful API.
All activity is logged asynchronously. If Elastic is down, calls to the API will fail with forbidden (403) errors. Once Elastic is available again, the API will function again.
All activity in internal services are logged (a?)synchronously. If Elastic is down the activity in internal services will still succeed, but there will be no logging. Future considerations may include changing to AMQ so as to not drop log entries if Elastic is down.
All activity on the the message queue for the router will be logged (a?)synchronously. This may or may not include the payload, depending on configuration.
Per Resource Indices
Each resource's log entries are kept in separate indices created either when the resource is created or when the resource is accessed, depending on the type of resource. Index names always start with "mdm-". This is used for both easy identification and to take advantage of Elastic's configuration ability to forbid index creation on the fly when the first data entry is posted to it. This allows the logging system to be in control of creating indices and mappings as needed.
Resource indices are created with the naming pattern "mdm-RESOURCE-UUID" where RESOURCE is the type of resource and, UUID is the zone's public facing UUID identifier. E.g. mdm-zone-a2aaedc7-591a-4761-8e35-da875b1e6ac5
Per User (AuthIdentity) Alias's
Alias's are created for the purposes of grouping indices together for ease of use in creating templates in Kibana for dashboards and views that are available through the YOUnite web application.
Alisas names are of the pattern mdm-alias-userUUID where userUUID is the public facing UUID identifier of the authIdentity.
What Gets Logged
The following items are currently logged for API access (see the table below for a complete list of logged events and what is logged):
- date/time
- API path
- resource type (matches the index type, e.g. mdm-zone is resource type zone, mdm-domain is resource type domain)
- resource UUID
- whether the access was allowed or rejected via OAuth
- ssoid of the OAuth token presented for access
- operation type (GET, PUT, DELETE, etc.)
The following items are currently logged for the resource service layer, which is internal, used by both the API and the message router, CRUD access to the resource:
- ssoid
- timestamp
- resource type (same as the index type)
- resource UUID
- action (CRUD)
The following items are currently logged for the Message Bus/router layer:
- Data Events
- API requests made to the /drs endpoint
- Callbacks generated by POST /drs/uuid/assembler
- Data events generated by adaptors
- Data event routing to adaptors
- Data event assembling
- API requests made to the /drs endpoint
- Adaptor registration
Either of these current logging entries can be easily extended to log additional request data (the object model being created, updated, etc.) or response data (results object model of a GET, etc.) for complete auditing of not only who made the modification, but what the modification was. This could be done for the entire resource spectrum or for only specific resources. If needed, the payload data could be easily logged to separate index requiring extended permissions for access.
TODO - PROVIDE TABLE OF WHAT IS LOGGED AND WHAT IS LOGGED ON A PER/ZONE BASIS VS ENTIRE ECOSYSTEM
How It Is Secured
Elastic and Kibana are secured via an NGINX proxy that authorizes each request via the OAuth. An OAuth bearer token is carried along on each request via a cookie when the user clicks the application dashboard link, displaying a Kibana-based view or dashboard page. This secures access to Elastic and Kibana from the public internet. If needed, the proxy piece could be integrated with Elastic in a single container to provide tighter security. This was not done in the dev environment for ease of development and testing, but could be done easily for a stand-alone Elastic server(s).
Kibana
When a user clicks a link to display logging information via Kibana, the URL is modified as it is passes through authorization code in the NGINX proxy. The modification uses the information in the OAuth check_token response to pass along the Elastic alias that Kibana will use to populate the dashboard's data. Access to Kibana is allowed if a user has the correct permissions for Kibana access. By default all users have this permission. Logging information that is displayed/available is controlled through the usage of either the alias for the user's UUID, which contains all the indices they have access to, or via Kibana searches hitting specific indices to which the user has direct permissions (also encoded in the OAuth token).
Elastic
Elastic is secured via the same NGINX proxy that protects Kibana. Permissions to allow read access to an index are controlled via the same permissions mechanism in the token used to allow a user's access to the YOUnite API. If a user has GET or ALL permissions to a zone, they have read-only access to the data in the index in Elastic for the same zone.
Logging Entriesfunctionality returns. In a production environment it is strongly recommended that Elastic is run as a highly-available, three-node cluster.
This page explains events that are logged and the fields and data values used in logging. As a YOUnite user, you can perform searches on requests and data events across the entire MDM ecosystem from a single dashboard.
Why Centralized Logging?
Having a centralized logging platform is critical in an MDM ecosystem because from a single dashboard, it allows:
- API consumers to debug and validate requests/responses
- Data Governance Stewards, DBAs, and similar data-focused staff to trace data origins and updates
- IT staff to track usage
What Gets Logged?
- All API requests made to YOUnite
- All data events:
- API requests made to the /drs endpoint
- Callbacks generated by POST /drs/uuid/assembler
- Data events generated by adaptors
- Data event routing to adaptors
- Data event assembling
- API requests made to the /drs endpoint
- Adaptor initialization
For a complete listing of what gets logged, the fields used including the data values that are used for specific fields - see the Logging Entries and Fields section below
IMPORTANT: Although YOUnite centrally logs all API requests and data events – for security reasons – it does NOT log the data payloads.
Kibana
There are two logging links in the upper right of the YOUnite UI header:
Image Added
- The first link takes the user to the Kibana Discover page where a user can filter and search on log entries based on "fields" (see Logging Entries and Fields) below.
- The second link takes the user to a Kibana dashboard customized for YOUnite.
Kibana Discover Page
All MDM events are logged using the fields defined in the Logging Entries and Fields section below.
Image Added
Kibana uses Lucene's query syntax in the search bar. Once you set the desired Time Range in the upper right corner you can query the central logs for MDM activity (go here for more on how to use the Discover Page). Following are some examples:
To see all GET data events on an adaptor with the UUID dd6e0bc6-b385-4c3b-b645-85ccd87c47e6:
Code Block | ||
---|---|---|
| ||
method:GET AND destination-adaptor-uuid:dd6e0bc6-b385-4c3b-b645-85ccd87c47e6 |
To trace a given data record (DR) with the UUID 621a5a25-e95a-475a-abbc-865da254522a:
Code Block | ||
---|---|---|
| ||
resource-uuid:621a5a25-e95a-475a-abbc-865da254522a |
To get all requests to the "/api/domains/*/versions" API endpoint and endpoints beneath it:
Code Block | ||
---|---|---|
| ||
request:"/api/domains/*/versions" |
Get all data events that attempted to generate new data records (the second example shows only the data events that succeeded):
Code Block | ||
---|---|---|
| ||
method:POST AND data-event-sequence:ORIGINATING AND log-entry-type:MB_REQUEST
method:POST AND data-event-sequence:ORIGINATING AND log-entry-type:MB_REQUEST AND status:MB_REQUEST_SUCCESS |
Same as above but restricted to a domain version (with UUID d8970cb7-bd6f-4371-8d66-9a66fc81d97f):
Code Block | ||
---|---|---|
| ||
method:POST AND data-event-sequence:ORIGINATING AND log-entry-type:MB_REQUEST AND status:MB_REQUEST_SUCCESS AND domain-version-uuid:d8970cb7-bd6f-4371-8d66-9a66fc81d97f |
Same as above but restricted to a given adaptor (with adaptor UUID 64c30e62-b363-4913-9e0a-0e6219b69eee)
Code Block | ||
---|---|---|
| ||
method:POST AND data-event-sequence:ORIGINATING AND log-entry-type:MB_REQUEST AND status:MB_REQUEST_SUCCESS AND domain-version-uuid:d8970cb7-bd6f-4371-8d66-9a66fc81d97f AND source-adaptor-uuid:64c30e62-b363-4913-9e0a-0e6219b69eee |
Anchor | ||||
---|---|---|---|---|
|
Image Added
Logging Entries and Fields
The table below is a summary of the what MDM events get logged and what values the values, or "fields," that get logged for each MDM event.
The following is a description of possible log entry data values:
REQUIRED
These required values are used to index log entries and are required:
- request-type: Requests are either directly related to DATA or other operational events designated as METADATA (see the "Request Types" table below). See the 949715042 list.
- resource-type: The API resource type as defined by the API endpoint (see the "Resource Types" table below). See the 949715042 list.
- zone-uuid: UUID of the zone the resource belongs touuid: UUID of the zone to which the resource belongs. For API requests, the UUID of the zone the resource belongs to. For adaptor or message bus requests, it's the UUID of the zone that owns the resource that generated the event.
- hostname: The hostname of the server handling and logging the request.
- entry-type: A general classification of for log entry types (see the "Audit Log Types" table below).
OPTIONAL
- method: The API method type called (See "Method Types" list).
- log-entry-type: Classification of log entry types (see "Log Entry Types" list).
- request: The URI of the API request. See the #entry-type list.
- timestamp: Timestamp of the event.
- zone-uuid: For API requests, the UUID of the zone the resource belongs to. For adaptor or message bus requests, its the UUID of the zone that owns the resource that generated the event. hostname: The hostname of the server handling and logging the
OPTIONAL
- method: The API method type called. See the #method list.
- log-entry-type: Classification of log entry types. See the #log-entry-type list.
- request: The URI of the API request.
- network-address: The network address of the server handling and logging the request.
- ssoid: The API consumer's single sign-on ID.
- auth-identity-uuid: The UUID of the API consumer (tied to the ssoid). Note that a single auth-identity can be tied to multiple zone users.
- http-status: The HTTP status code of the API request.
- data-event-sequence: Federated data events go through a sequence of one or more of these steps:
- Originating data event
- Routing data events to adaptors
- Receiving responses from adaptors for routed data events
- Assembling responses
See the #data-event-sequence list.
- dr-notification-uuid: The UUID of the assembler object. An assembler object is created for requests that pull data from one or more adaptors.data-event-uuid: A single originating data-event typically generates multiple data-events, each with its own "data-event-uuid", for all appropriate adaptors when it needs to assemble datadata from one or more adaptors.
- data-event-uuid: A When it needs to assemble data, a single originating data-event typically generates multiple data-events, each with its own "data-event-uuid", for all appropriate adaptors when it needs to assemble data from or broadcast data out to adaptors.
- source-adaptor-uuid: The adaptor from which a data-event originated from.
- destination-adaptor-uuid: The adaptor to which a data-event is getting sent to.
- adaptor-capabilities: List A list of the adaptor's domain version property-processing capabilities.
- adaptor-state: The adaptor's state (see "Adaptor States" list). See the #adaptor-state list.
- domain-version-uuid: The domain version specified by a data event.
- resource-uuid: The UUID of the resource the request is operating on.
- request-uuid: API requests are given a random request UUID.ssoid: The API consumer's single sign-on ID.
- gold-adaptor-uuids: A list of adaptors designated as "gold" in the data request.
- silver-adaptor-uuids: A list of adaptors designated as "silver" in the data request.
- status: Status for a non-HTTP event. See the #status list.
- message: Typically, but not limited to, an error message.
The values on the left table below are used to index log entries and are required. The right side table below shows what which values are logged for the various MDM events.
REQUIRED: DataValuesFields Used to Index Log Entries | OPTIONAL: Log Entry Field Data Values | ||||||||||||||||||||||||||||
MDM Events | request-type | resource-type | zone-uuid | entry-type | method | log-entry-type | request | timestamp | zone-uuid | hostname | network-address | auth-identity-uuid | http-status | data-event-seqeuence | dr-notification-uuid | data-event-uuid | source-adaptor-uuid | destination-adaptor-uuid | adaptor-capabilities | adaptor-state | domain-version-uuid | resource-uuid | request-uuid | ssoid | gold-adaptor-uuids | silver-adaptor-uuids | status | message | |
API Request (L20) | METADATA | yes | yes | API | yes | API_REQUEST | yes | yes | yes | yes | yes | yes | yes | yes | |||||||||||||||
Data API Request (L25) | DATA | Limited to DRS or DRS_ASSEMBLER | yes | API | yes | API_REQUEST | yes | yes | yes | yes | yes | yes | yes (dr-uuid) | yes | yes | ||||||||||||||
API Response (L30) | METADATA | yes | yes | API | yes | API_REQUEST_SUCCESS | yes | yes | yes | yes | yes | yes | yes | yes | yes | ||||||||||||||
Data API Response (L35) | DATA | Limited to DRS or DRS_ASSEMBLER | yes | API | yes | API_REQUEST_SUCCESS | yes | yes | yes | yes | yes | yes | yes | yes (dr-uuid) | yes | yes | |||||||||||||
API Error Response (L40) | METADATA | yes | yes | API | yes | API_REQUEST_ERROR | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | |||||||||||||
Data API Error Response (L45) | DATA | yes | yes | API | yes | API_REQUEST_ERROR | yes | yes | yes | yes | yes | yes | yes | yes (dr-uuid) | yes | yes | yes | ||||||||||||
API Request Security Violation (L100) | METADATA | yes | yes | API | yes | API_REQUEST_ERROR | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | |||||||||||||
MDM Events | request-type | resource-type | zone-uuid | entry-type | method | log-entry-type | request | timestamp | zone-uuid | hostname | network-address | auth-identity-uuid | http-status | data-event-seqeuence | dr-notification-uuid | data-event-uuid | source-adaptor-uuid | destination-adaptor-uuid | adaptor-capabilities | adaptor-state | domain-version-uuid | resource-uuid | request-uuid | ssoid | gold-adaptor-uuids | silver-adaptor-uuids | status | message | |
Service Layer POST Assembler Processing (L1) | DATA | DRS_ASSEMBLER | yes | ADAPTOR | GET1 | MB_REQUEST | yes | yes | yes | yes | ORIGINATING | yes | yes | yes (dr-uuid) | yes | optional | optional | MB_REQUEST_SUCCESS or MB_REQUEST_ERROR | yes (if there is an error) | ||||||||||
POST Assembler Routing Events (L2) | DATA | DRS_ASSEMBLER | yes | ADAPTOR | GET1 | MB_REQUEST | yes | yes | yes | yes | ROUTED | yes | yes | yes | yes | yes (dr-uuid) | MB_REQUEST_SUCCESS or MB_REQUEST_ERROR | yes (if there is an error) | |||||||||||
POST Assembler Routing Event Returned (L3) | DATA | DRS_ASSEMBLER | yes | ADAPTOR | GET1 | MB_REQUEST | yes | yes | yes | yes | ROUTED_RETURNED | yes | yes | yes | yes | yes (dr-uuid) | MB_REQUEST_SUCCESS or MB_REQUEST_ERROR | yes (if there is an error) | |||||||||||
POST Assembler Callback (L4) | DATA | DRS_ASSEMBLER | yes | ADAPTOR | GET1 | MB_REQUEST | yes | yes | yes | yes | ASSEMBLED | yes | yes | yes | yes | yes (dr-uuid) | MB_REQUEST_SUCCESS or MB_REQUEST_ERROR | yes (if there is an error) | |||||||||||
MDM Events | request-type | resource-type | zone-uuid | entry-type | method | log-entry-type | request | timestamp | zone-uuid | hostname | network-address | auth-identity-uuid | http-status | data-event-seqeuence | dr-notification-uuid | data-event-uuid | source-adaptor-uuid | destination-adaptor-uuid | adaptor-capabilities | adaptor-state | domain-version-uuid | resource-uuid | request-uuid | ssoid | gold-adaptor-uuids | silver-adaptor-uuids | status | message | |
PUT, POST, DELETE Event Received From Adaptor (L5) | DATA | DRS_ASSEMBLER | yes | ADAPTOR | PUT, POST or DELETE | MB_REQUEST | yes | yes | yes | yes | ORIGINATING | yes | yes | yes | yes | yes (dr-uuid for PUT and DELETE and null for POST) | MB_REQUEST_SUCCESS or MB_REQUEST_ERROR | yes (if there is an error) | |||||||||||
PUT, POST, DELETE Event Routed to Adaptor (L6) | DATA | DRS_ASSEMBLER | yes | ADAPTOR | PUT, POST or DELETE | MB_REQUEST | yes | yes | yes | yes | ROUTED | yes | yes | yes | yes | yes | yes (dr-uuid) | MB_REQUEST_SUCCESS or MB_REQUEST_ERROR | yes (if there is an error) | ||||||||||
Adaptor Registration (L10) | METADATA | ADAPTORS | yes | ADAPTOR | ADAPTOR_STATE | MB_REQUEST | yes | yes | yes | yes | yes | yes | yes (See "ADAPTOR STATES" list) | MB_REQUEST_SUCCESS or MB_REQUEST_ERROR | yes (if there is an error) |
1 POST /drs/uuid for a FEDERATED domain version is used in place of a GET /drs/uuid (used for an MDM_DATA_STORE domain version) since assembling a data record takes extra processing – but logically processing. Logically it is a GET, and governance permissions treat POST /drs/uuid as a GET.
Request TypesData Field Values
Section | ||||||||
---|---|---|---|---|---|---|---|---|
|
ZONES
|
|
|
Section | |||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
|
|
|
Image Removed
Notifications
For more information goto
Highlevel discussion on what sort of notifications get sent out and how we leverage webhooks.Image Modified