Logging

YOUnite Uses Elastic

YOUnite uses the Open Source Elastic Stack (https://www.elastic.co/products) to log (Logstash), search (Elasticsearch), analyze and visualize (Kibana) all API requests and message bus traffic (most notably all data events) .

YOUnite logs directly to Elastic via the Elastic RESTtful API.

All activity is logged asynchronously. If Elastic is down, calls to the API will fail with forbidden (403) errors. Once Elastic is available again, the API will function again. In a production environment it is strongly recommended that a Elastic is run as a highly available three-node cluster.

This page covers what gets logged and provides the information on fields and data values used so that a YOUnite user can perform searches on requests and data events across the entire MDM ecosystem from a single dashboard.

What Gets Logged

Why Centralized Logging?

Having a centralized logging platform is critical in an MDM ecosystem because from a single dashboard, it allows:

• API consumers to debug and validate requests/responses
• Data Governance, DBAs and similar data focused staff to trace data origins and updates
• IT staff to track usage

What Gets Logged?

• All API requests made to YOUnite
• All data events:
  • API requests made to the /drs endpoint
    • Callbacks generated by POST /drs/uuid/assembler 
  • Data events generated by adaptors
  • Data event routing to adaptors
  • Data event assembling
• Adaptor initialization

For a complete listing of what gets logged, the fields used including the data values that are used for specific fields - see the “Logging Entries and Fields
” section belowEither of these current logging entries can be easily extended to log additional request data (the object model being created, updated, etc.) or response data (results object model of a GET, etc.) for complete auditing of not only who made the modification, but what the modification was. This could be done for the entire resource spectrum or for only specific resources. If needed, the payload data could be easily logged to separate index requiring extended permissions for access

IMPORTANT: Although YOUnite centrally logs all API requests and data events – for security reasons -- it does NOT log the data payloads.

Kibana

There are two logging links in the YOUnite UI header:

• The first link takes the user to the Kibana "Discover" page where a user can filter and search on log entries based on "fields" (see "Logging Entries and Fields") below.
• The second link takes the user to a Kibana dashboard customized for YOUnite.

Discover Page

All MDM events are logged using the fields defined in the "Logging Entries and Fields" section below. 

Kibana uses Lucene's query syntax. For example, to see all GET data events for a given adaptor:

method:GET AND destination-adaptor-uuid:dd6e0bc6-b385-4c3b-b645-85ccd87c47e6

Dashboard

Logging Entries and Fields

The table below is a summary of the what MDM events get logged and what values or "fields" get logged for each MDM event.

The following is a description of possible log entry data values:

REQUIRED

These values are used to index log entries and are required:

• request-type: Requests are either directly related to DATA or other operational events designated as METADATA (see the "Request Types" table below).
• resource-type: The API resource type as defined by the API endpoint (see the "Resource Types" table below).
• zone-uuid: UUID of the zone the resource belongs to.
• entry-type: A general classification of for log entry types (see the "Audit Log Types" table below).

OPTIONAL

• method: The API method type called (See "Method Types" list).
• log-entry-type: Classification of log entry types (see "Log Entry Types" list).
• request: The URI of the API request.
• timestamp: Timestamp of the event.
• zone-uuid: For API requests, the UUID of the zone the resource belongs to. For adaptor or message bus requests, its the UUID of the zone that owns the resource that generated the event.
• hostname: The hostname of the server handling and logging the request.
• network-address: The network address of the server handling and logging the request.
• ssoid: The API consumer's single sign-on ID.
• auth-identity-uuid: The UUID of the API consumer (tied to the ssoid). Note that a single auth-identity can be tied to multiple zone users.
• http-status: The HTTP status code of the API request.
• data-event-sequence: Federated data events go through a sequence of one or more of these steps:
  • Originating data event
  • Routing data events to adaptors
  • Receiving responses from adaptors for routed data events
  • Assembling responses 
• dr-notification-uuid: The UUID of the assembler object. An assembler object is created for requests that pull data from one or more adaptors.
• data-event-uuid: A single originating data-event typically generates multiple data-events, each with its own "data-event-uuid", for all appropriate adaptors when it needs to assemble data
• data-event-uuid: A single originating data-event typically generates multiple data-events, each with its own "data-event-uuid", for all appropriate adaptors when it needs to assemble data from or broadcast data out to adaptors
• source-adaptor-uuid: The adaptor a data-event originated from.
• destination-adaptor-uuid: The adaptor a data-event is getting sent to.
• adaptor-capabilities: List of the adaptor's domain version property processing capabilities.
• adaptor-state: The adaptor's state (see "Adaptor States" list).
• domain-version-uuid: The domain version specified by a data event.
• resource-uuid: The UUID of the resource the request is operating on.
• request-uuid: API requests are given a random request UUID.
• ssoid: The API consumer's single sign-on ID.
• gold-adaptor-uuids: A list of adaptors designated as gold in the data request.
• silver-adaptor-uuids: A list of adaptors designated as silver in the data request.
• status: Status for a non HTTP event.
• message: Typically, but not limited to,  an error message.

The values on the left are used to index log entries and are required. The right side shows what values are logged for the various MDM events.

¹ POST /drs/uuid for a FEDERATED domain version is used in place of a GET /drs/uuid (used for an MDM_DATA_STORE domain version) since assembling a data record takes extra processing – but logically it is a GET and governance permissions treat POST /drs/uuid as a GET.

Data Field Values