Configuring logging is covered Zone Management: Logging page.
Former user (Deleted) We might want to break this out into two pages.
Logging
Table of Contents |
---|
YOUnite Uses Elastic
YOUnite logs directly to Elastic via the Elastic RESTtful API.
All activity through the MDM RESTful API is logged synchronously. If Elastic is down, calls to the API will fail with forbidden (403) errors. Once Elastic is available again, the API will function again.
All activity in internal services are logged (a?)synchronously. If Elastic is down the activity in internal services will still succeed, but there will be no logging. Future considerations may include changing to AMQ so as to not drop log entries if Elastic is down.
All activity on the the message queue for the router will be logged (a?)synchronously. This may or may not include the payload, depending on configuration.
Per Resource Indices
Each resource's log entries are kept in separate indices created either when the resource is created or when the resource is accessed, depending on the type of resource. Index names always start with "mdm-". This is used for both easy identification and to take advantage of Elastic's configuration ability to forbid index creation on the fly when the first data entry is posted to it. This allows the logging system to be in control of creating indices and mappings as needed.
Resource indices are created with the naming pattern "mdm-RESOURCE-UUID" where RESOURCE is the type of resource and, UUID is the zone's public facing UUID identifier. E.g. mdm-zone-a2aaedc7-591a-4761-8e35-da875b1e6ac5
Per User (AuthIdentity) Alias's
Alias's are created for the purposes of grouping indices together for ease of use in creating templates in Kibana for dashboards and views that are available through the YOUnite web application.
Alisas names are of the pattern mdm-alias-userUUID where userUUID is the public facing UUID identifier of the authIdentity.
What Gets Logged
The following items are currently logged for API access (see the table below for a complete list of logged events and what is logged):
- date/time
- API path
- resource type (matches the index type, e.g. mdm-zone is resource type zone, mdm-domain is resource type domain)
- resource UUID
- whether the access was allowed or rejected via OAuth
- ssoid of the OAuth token presented for access
- operation type (GET, PUT, DELETE, etc.)
The following items are currently logged for the resource service layer, which is internal, used by both the API and the message router, CRUD access to the resource:
- ssoid
- timestamp
- resource type (same as the index type)
- resource UUID
- action (CRUD)
The following items are currently logged for the Message Bus/router layer:
- Data Events
- API requests made to the /drs endpoint
- Callbacks generated by POST /drs/uuid/assembler
- Data events generated by adaptors
- Data event routing to adaptors
- Data event assembling
- API requests made to the /drs endpoint
- Adaptor registration
Either of these current logging entries can be easily extended to log additional request data (the object model being created, updated, etc.) or response data (results object model of a GET, etc.) for complete auditing of not only who made the modification, but what the modification was. This could be done for the entire resource spectrum or for only specific resources. If needed, the payload data could be easily logged to separate index requiring extended permissions for access.
TODO - PROVIDE TABLE OF WHAT IS LOGGED AND WHAT IS LOGGED ON A PER/ZONE BASIS VS ENTIRE ECOSYSTEM
How It Is Secured
Elastic and Kibana are secured via an NGINX proxy that authorizes each request via the OAuth. An OAuth bearer token is carried along on each request via a cookie when the user clicks the application dashboard link, displaying a Kibana-based view or dashboard page. This secures access to Elastic and Kibana from the public internet. If needed, the proxy piece could be integrated with Elastic in a single container to provide tighter security. This was not done in the dev environment for ease of development and testing, but could be done easily for a stand-alone Elastic server(s).
Kibana
There are two logging links in the YOUnite UI header:
- The first link takes the user to the Kibana "Discover" page where a user can filter and search on log entries based on "fields" (see "Logging Entries and Fields") below.
- The second link takes the user to a Kibana dashboard customized for YOUnite.
Discover Page
All MDM events are logged using the fields defined in the "Logging Entries and Fields" section below.
Kibana uses Lucene's query syntax. For example, to see all GET data events for a given adaptor:
Code Block | ||
---|---|---|
| ||
method:GET AND destination-adaptor-uuid:dd6e0bc6-b385-4c3b-b645-85ccd87c47e6 |
Dashboard
Background on How the YOUnite UI Makes the Dashboard Connection to Kibana
When a user clicks a link to display logging information via Kibana, the URL is modified as it is passes through authorization code in the NGINX proxy. The modification uses the information in the OAuth check_token response to pass along the Elastic alias that Kibana will use to populate the dashboard's data. Access to Kibana is allowed if a user has the correct permissions for Kibana access. By default all users have this permission. Logging information that is displayed/available is controlled through the usage of either the alias for the user's UUID, which contains all the indices they have access to, or via Kibana searches hitting specific indices to which the user has direct permissions (also encoded in the OAuth token).