YOUnite Logging


Configuring logging is covered  Deploying Logging page.


Logging

YOUnite Uses Elastic

YOUnite uses the Open Source Elastic Stack (https://www.elastic.co/products) to log (Logstash), search (Elasticsearch), analyze and visualize (Kibana) all API requests and message bus traffic (most notably all data events).

YOUnite logs directly to Elastic via Elastic's RESTtful API.

All activity is logged asynchronously. If Elastic is down, calls to the API fail with forbidden (403) errors. Once Elastic is available again, API functionality returns. In a production environment it is strongly recommended that Elastic is run as a highly-available, three-node cluster.

This page explains events that are logged and the fields and data values used in logging. As a YOUnite user, you can perform searches on requests and data events across the entire MDM ecosystem from a single dashboard.

Why Centralized Logging?

Having a centralized logging platform is critical in an MDM ecosystem because from a single dashboard, it allows:

  • API consumers to debug and validate requests/responses
  • Data Governance Stewards, DBAs, and similar data-focused staff to trace data origins and updates
  • IT staff to track usage

What Gets Logged?

  • All API requests made to YOUnite
  • All data events:
    • API requests made to the /drs endpoint
      • Callbacks generated by POST /drs/uuid/assembler 
    • Data events generated by adaptors
    • Data event routing to adaptors
    • Data event assembling
  • Adaptor initialization

For a complete listing of what gets logged, the fields used including the data values that are used for specific fields - see the Logging Entries and Fields section below



IMPORTANT: Although YOUnite centrally logs all API requests and data events – for security reasons – it does NOT log the data payloads.


Kibana

There are two logging links in the upper right of the YOUnite UI header:



Kibana Discover Page

All MDM events are logged using the fields defined in the Logging Entries and Fields section below. 


Kibana uses Lucene's query syntax in the search bar. Once you set the desired Time Range in the upper right corner you can query the central logs for MDM activity (go here for more on how to use the Discover Page).   Following are some examples:

To see all GET data events on an adaptor with the UUID dd6e0bc6-b385-4c3b-b645-85ccd87c47e6:

method:GET AND destination-adaptor-uuid:dd6e0bc6-b385-4c3b-b645-85ccd87c47e6

To trace a given data record (DR) with the UUID 621a5a25-e95a-475a-abbc-865da254522a:

resource-uuid:621a5a25-e95a-475a-abbc-865da254522a

To get all requests to the "/api/domains/*/versions" API endpoint and endpoints beneath it:

request:"/api/domains/*/versions"

Get all data events that attempted to generate new data records (the second example shows only the data events that succeeded):

method:POST AND data-event-sequence:ORIGINATING AND log-entry-type:MB_REQUEST
method:POST AND data-event-sequence:ORIGINATING AND log-entry-type:MB_REQUEST AND status:MB_REQUEST_SUCCESS 

Same as above but  restricted to a domain version (with UUID d8970cb7-bd6f-4371-8d66-9a66fc81d97f):

method:POST AND data-event-sequence:ORIGINATING AND log-entry-type:MB_REQUEST AND status:MB_REQUEST_SUCCESS AND domain-version-uuid:d8970cb7-bd6f-4371-8d66-9a66fc81d97f

Same as above but restricted to a given adaptor (with adaptor UUID 64c30e62-b363-4913-9e0a-0e6219b69eee)

method:POST AND data-event-sequence:ORIGINATING AND log-entry-type:MB_REQUEST AND status:MB_REQUEST_SUCCESS AND domain-version-uuid:d8970cb7-bd6f-4371-8d66-9a66fc81d97f AND source-adaptor-uuid:64c30e62-b363-4913-9e0a-0e6219b69eee

Kibana Dashboard

Logging Entries and Fields

The table below is a summary of the MDM events and the values, or "fields," that get logged for each MDM event.

The following is a description of possible log entry data values:

REQUIRED

These required values are used to index log entries:

  • request-type: Requests are either directly related to DATA or other operational events designated as METADATA. See the YOUnite Logging#request-type list.
  • resource-type: The API resource type as defined by the API endpoint. See the YOUnite Logging#resource-type list.
  • zone-uuid: UUID of the zone to which the resource belongs. For API requests, the UUID of the zone the resource belongs to. For adaptor or message bus requests, it's the UUID of the zone that owns the resource that generated the event.
  • hostname: The hostname of the server handling and logging the request.
  • entry-type: A general classification of for log entry types. See the #entry-type list.
  • timestamp: Timestamp of the event.

OPTIONAL

  • method: The API method type called. See the #method list.
  • log-entry-type: Classification of log entry types. See the #log-entry-type list.
  • request: The URI of the API request.
  • network-address: The network address of the server handling and logging the request.
  • ssoid: The API consumer's single sign-on ID.
  • auth-identity-uuid: The UUID of the API consumer (tied to the ssoid). Note that a single auth-identity can be tied to multiple zone users.
  • http-status: The HTTP status code of the API request.
  • data-event-sequence: Federated data events go through a sequence of one or more of these steps:
    • Originating data event
    • Routing data events to adaptors
    • Receiving responses from adaptors for routed data events
    • Assembling responses 

See the #data-event-sequence list.

  • dr-notification-uuid: The UUID of the assembler object. An assembler object is created for requests that pull data from one or more adaptors.
  • data-event-uuid: When it needs to assemble data, a single originating data-event typically generates multiple data-events, each with its own "data-event-uuid", for all appropriate adaptors.
  • source-adaptor-uuid: The adaptor from which a data-event originated.
  • destination-adaptor-uuid: The adaptor to which a data-event is sent.
  • adaptor-capabilities: A list of the adaptor's domain version property-processing capabilities.
  • adaptor-state: The adaptor's state. See the #adaptor-state list.
  • domain-version-uuid: The domain version specified by a data event.
  • resource-uuid: The UUID of the resource the request is operating on.
  • request-uuid: API requests are given a random request UUID.
  • gold-adaptor-uuids: A list of adaptors designated as "gold" in the data request.
  • silver-adaptor-uuids: A list of adaptors designated as "silver" in the data request.
  • status: Status for a non-HTTP event. See the #status list.
  • message: Typically, but not limited to, an error message.


The values on the left table below are used to index log entries and are required. The right table below shows which values are logged for the various MDM events.


REQUIRED: Data Fields Used to Index Log Entries
OPTIONAL:  Log Entry Field Data Values
MDM Events   request-typeresource-typezone-uuidentry-type
methodlog-entry-typerequesttimestampzone-uuidhostnamenetwork-addressauth-identity-uuidhttp-statusdata-event-seqeuencedr-notification-uuiddata-event-uuidsource-adaptor-uuiddestination-adaptor-uuidadaptor-capabilitiesadaptor-statedomain-version-uuidresource-uuidrequest-uuidssoidgold-adaptor-uuidssilver-adaptor-uuidsstatusmessage

API Request (L20)

METADATAyesyesAPIyesAPI_REQUESTyesyesyesyesyesyes









yesyes



Data API Request (L25)DATALimited to DRS or DRS_ASSEMBLERyesAPIyesAPI_REQUESTyesyesyesyesyesyes








yes (dr-uuid)yesyes



API Response (L30)METADATAyesyesAPIyesAPI_REQUEST_SUCCESSyesyesyesyesyesyesyes








yesyes



Data API Response (L35)DATALimited to DRS or DRS_ASSEMBLERyesAPIyesAPI_REQUEST_SUCCESSyesyesyesyesyesyesyes







yes (dr-uuid)yesyes



API Error Response (L40)

METADATAyesyesAPIyesAPI_REQUEST_ERRORyesyesyesyesyesyes yes








 yes yes


yes
Data API Error Response (L45)DATAyesyesAPIyesAPI_REQUEST_ERRORyesyesyesyesyesyes yes







yes (dr-uuid) yes yes


yes
API Request Security Violation (L100)METADATAyesyesAPIyesAPI_REQUEST_ERRORyesyesyesyesyesyesyes








yesyes


yes


MDM Events   request-typeresource-typezone-uuidentry-typemethodlog-entry-typerequesttimestampzone-uuidhostnamenetwork-addressauth-identity-uuidhttp-statusdata-event-seqeuencedr-notification-uuiddata-event-uuidsource-adaptor-uuiddestination-adaptor-uuidadaptor-capabilitiesadaptor-statedomain-version-uuidresource-uuidrequest-uuidssoidgold-adaptor-uuidssilver-adaptor-uuidsstatusmessage
Service Layer POST Assembler Processing (L1)DATADRS_ASSEMBLERyesADAPTORGET1MB_REQUEST
yesyesyesyes

ORIGINATINGyesyes




yes (dr-uuid)
yesoptionaloptionalMB_REQUEST_SUCCESS or MB_REQUEST_ERRORyes (if there is an error)
POST Assembler Routing Events (L2)DATADRS_ASSEMBLERyesADAPTORGET1MB_REQUEST
yesyesyesyes

ROUTEDyesyes
yes

yesyes (dr-uuid)



MB_REQUEST_SUCCESS or MB_REQUEST_ERRORyes (if there is an error)
POST Assembler Routing Event Returned (L3)DATADRS_ASSEMBLERyesADAPTORGET1MB_REQUEST
yesyesyesyes

ROUTED_RETURNEDyesyesyes


yesyes (dr-uuid)



MB_REQUEST_SUCCESS or MB_REQUEST_ERROR

yes (if there is an error)
POST Assembler Callback (L4)DATADRS_ASSEMBLERyesADAPTORGET1MB_REQUEST
yesyesyesyes

ASSEMBLEDyesyesyes


yesyes (dr-uuid)



MB_REQUEST_SUCCESS or MB_REQUEST_ERRORyes (if there is an error)


MDM Events  request-typeresource-typezone-uuidentry-typemethodlog-entry-typerequesttimestampzone-uuidhostnamenetwork-addressauth-identity-uuidhttp-statusdata-event-seqeuencedr-notification-uuiddata-event-uuidsource-adaptor-uuiddestination-adaptor-uuidadaptor-capabilitiesadaptor-statedomain-version-uuidresource-uuidrequest-uuidssoidgold-adaptor-uuidssilver-adaptor-uuidsstatusmessage
PUT, POST, DELETE Event Received From Adaptor (L5)DATADRS_ASSEMBLERyesADAPTORPUT, POST or DELETEMB_REQUEST
yesyesyesyes

ORIGINATINGyesyesyes


yesyes (dr-uuid for PUT and DELETE and null for POST)



MB_REQUEST_SUCCESS or MB_REQUEST_ERRORyes (if there is an error)
PUT, POST, DELETE Event Routed to Adaptor (L6)DATADRS_ASSEMBLERyesADAPTORPUT, POST or DELETEMB_REQUEST
yesyesyesyes

ROUTEDyesyesyesyes

yesyes (dr-uuid)



MB_REQUEST_SUCCESS or MB_REQUEST_ERRORyes (if there is an error)


Adaptor Registration (L10)METADATAADAPTORSyesADAPTORADAPTOR_STATEMB_REQUEST
yesyesyesyes




yes
yesyes (See "ADAPTOR STATES" list)





MB_REQUEST_SUCCESS or MB_REQUEST_ERRORyes (if there is an error)

1 POST /drs/uuid for a FEDERATED domain version is used in place of a GET /drs/uuid (used for an MDM_DATA_STORE domain version) since assembling a data record takes extra processing. Logically it is a GET, and governance permissions treat POST /drs/uuid as a GET.

Data Field Values


request-type
METADATA
DATA
resource-type       

ZONES                  

DOMAINS             

SCHEMAS

DRS

ACLS

ADAPTORS

RESOURCES

DRS_ASSEMBLER

entry-type
API
ADAPTOR
MB
SERVICE
method

GET

PUT
POST
DELETE
PATCH
ADAPTOR_STATE
log-entry-type

API_REQUEST

API_REQUEST_SUCCESS

API_REQUEST_ERROR

API_WEBSOCKET

API_WEBSOCKET_SUCCESS

API_WEBSOCKET_ERROR

API_WEBHOOK

API_WEBHOOK_SUCCESS

API_WEBHOOK_ERROR

MB_REQUEST

MB_REQUEST_SUCCESS

MB_REQUEST_ERROR

MB_RESPONSE

MB_RESPONSE_SUCCESS

MB_RESPONSE_ERROR

status

MB_REQUEST_SUCCESS

MB_REQUEST_ERROR

MB_RESPONSE_SUCCESS

MB_RESPONSE_ERROR

data-event-sequence

ORIGINATING

ROUTED

ROUTED_RETURNED

ASSEMBLED

adaptor-state
ADAPTOR_POSTED
ADAPTOR_ERROR
ADAPTOR_INVALID_CAPABILITIES
ADAPTOR_CONFIGURES
ADAPTOR_PAUSE
ADAPTOR_PLAY
ADAPTOR_PLAY_RO (READY ONLY)