...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
| panelIconId | atlassian-check_mark |
|---|---|
| panelIcon | :check_mark: |
| bgColor | #F4F5F7 |
...
Introduction
This guide provides an overview of the implementation configuration for establishing a direct access connection (ODBC/JDBC) to the CCC Data Warehouse for authorized California Community Colleges (CCC) districts and colleges
...
.
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Secure access to CCC Data is provided to institutional researchers, college and district administrators, and other decision makers at the California Community Colleges, district offices, and the Chancellor’s Office, where these critical data may be used to support instructional and institutional decision-making aligned with the Chancellor's Office Vision 2030. |
...
Contents
| Table of Contents | ||
|---|---|---|
|
...
About the CCC Data Warehouse
The CCC Data Warehouse is developed by the CCC Technology Center (CCCTC) in coordination with, and at the direction of, the CCC Chancellor's Office. A part of the Data Services Program initiative from the California Community Colleges Chancellor's Office, the CCC Data Warehouse provides the necessary infrastructure to the California Community College System to aggregate data from across disparate systems to an enterprise data warehouse.
Direct Access Connection
For authorized Researchers researchers at CCC institutions who want to connect local applications to data sources within CCC Data, the CCC Tech Center ( CCCTC ) now supports direct access connection to the CCC Data Warehouse as an alternate or additional option to the CCC Data Warehouse (Jaspersoft) Report Server. Local Local applications at CCC colleges/districts may include an analytics or business intelligence application such as Tableau or Power BI.
Once connected to the CCC Data Warehouse, your college or district will be able to:
Connect to available data sources through a site-to-site VPN using service account login credentials (provided).
Run SQL queries against available data sources .
Connect a local analytics/BI application through ODBC or JDBC access.
Connect CCC Data to a /your local data warehouse by setting up these data as external tables within your local data warehouse.
| Panel | ||||||
|---|---|---|---|---|---|---|
| Note | ||||||
| ||||||
IMPORTANT: The CCC Data Warehouse is should not to be used as a staging ground for data distribution to your local data warehouse. This type of usage will not be permitted, as it places an undue load and cost on the CCC Data infrastructure. | ||||||
| Panel | ||||||
| ||||||
Reminder: Direct access is an alternate or additional option to the CCC Data Warehouse Report Server for authorized CCC institutional researchers. |
Available Data Sources
Direct access connection is supported for the following data sources within the CCC Data Warehouse:
...
Table A: Data Availability Chart
Source Data Set | CCCCO Access | College Access | Requires Opt-In for inclusion | ||
|---|---|---|---|---|---|
DW Direct | DW | DW Direct | DW | ||
CCCApply Standard Application |
|
|
|
| |
CCCApply Noncredit Application |
|
|
|
| |
CCCApply International Application |
|
|
|
| |
California College Promise Grant Application |
|
|
|
| |
Decrypted LGBTQ Data (and Report) |
|
| |||
|
| ||||
Multiple Measures Placement Service (MMPS) |
|
|
|
| |
MyPath |
|
|
|
| |
COCI: Chancellor's Office Curriculum Inventory |
|
|
|
| |
C-ID: Course ID Numbering System |
|
|
|
| |
Canvas Data 2 (Opt-in required) |
|
|
| |
CO-MIS |
|
NOVA |
| ||||
Launchboard |
|
CTEOS |
|
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Data dictionaries for the above data sources are available on the CCC Data Warehouse documentation site. |
...
Implementation Process
Upon When a request is received from an authorized individual at a college or district office (often the Dean overseeing Institutional Researchinstitutional research), one or more members of the CCCTC Enabling Services (ES) team will work directly with your the college IT group to configure a custom site-to-site VPN based on your system, and establish the system used by the college. Enabling Services will then establish a secure direct access connection to the CCC Data Warehouse from your the college (or district). Following the After successful implementation and query validations, the ES Support team will be able to provide post-implementation (Livelive) support through the CCCTechnology.info support channel.
...
The process to implement a direct access connection to the CCC Data Warehouse (Redshift) is described below, including beginning with a summary of the roles and responsibilities for participants, is listed below.
...
.
...
Participants, Roles, and Responsibilities
Participant | Is responsible for… |
|---|---|
College/District Researcher |
|
Enabling Services (ES) College Relationship Manager (CEM) |
|
ES Implementation Configuration Engineer (ICE) |
|
College/District IT/Network Admin |
|
ES Implementation Configuration Engineer (ICE) |
|
ES Support Services |
|
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
IMPORTANT: Colleges and districts using that use their own analytics/BI applications are responsible for their own support in the use of these using those tools. | ||||||
| Panel | ||||||
|
...
Request Access
The following are the general steps to establish a direct access connection (ODBC/JDBC) to the CCC Data Warehouse.
...
An authorized individual at the college or district
...
makes the request directly to the CEM assigned to
...
the college. This request is typically made by the Vice Chancellor or the Dean overseeing institutional research.
Upon approval of the request, the CEM initiates the implementation process by passing the required information to
...
Enabling Services (ES).
The ES implementation team
...
then creates the college’s custom VPN and sets up the account credentials.
...
Provide the following information to
...
the CEM when requesting direct access:
...
-Contact
...
name and email address of the
...
primary researcher responsible for the account login credentials
...
-Contact name and email address of
...
primary IT/
...
network admin who will work with ES to establish
...
site-to-site VPN connection
...
...
-System configuration information such as firewall make, model, and version,
...
as well as IP information
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Start with Your CEMNote: The initial direct connection request should be made to your College Relationship Experience Manager. If you do not know who your CEM ishave the name of the CEM for your college, send an email to cems@ccctechcenter.org. |
...
| panelIconId | atlassian-check_mark |
|---|---|
| panelIcon | :check_mark: |
| bgColor | #F4F5F7 |
...
Establish VPN Configuration
The ES Implementation Configuration Engineer (ICE) works with your primary IT contact(s) to facilitate the configuration of the VPN and to establish the account credentials.
Establishing the VPN Configuration
A secure direct access (ODBC/JDBC) connection between your college or district (MIS scope) and the CCC Data Warehouse Redshift cluster is enabled using a custom site-to-site VPN tunnel and a service account with authorized user credentials. The data that is accessible by the college or district is based on the scope of access authorized to the organization.
Setting up the VPN and the service account can happen occur simultaneously; however the configuration of the VPN should happen be handled first, as it often requires more time to implement (due to required approvals , and IT resources).
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
ReminderNote: Ensure your that the ES ICE engineer has all necessary system configuration information, including your the firewall make, model, and version, and your as well as IP information. |
...
| panelIconId | atlassian-check_mark |
|---|---|
| panelIcon | :check_mark: |
| bgColor | #F4F5F7 |
...
Establish Account Credentials
In order to access the (Redshift) databases, a service account will must be established based on the scope of your authorized access (MIS scope). Your The ES ICE engineer will provide establishes the service account and provides the account login credentials (username and password), as well as the to the primary IT contact at the college/district. (The login credentials consist of the username and password.)
The ES ICE also provides the database endpoint (DNS) and schemas roles (schema roles to the college or district’s primary IT contact via secure transmission. (The schema roles are also provided in this document), to your primary IT contact through secure transmission. .)
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Note that the : The login credentials are a Service Account, while . While they are issued to an individual, they are intended to be used to connect applications (such as Tableau) to the CCC Data Warehouse. Protecting these credentials is the responsibility of the college/district. |
Scope of Access
Authorized access to the CCC Data Warehouse for colleges and districts using direct access connection (ODBC/JDBC) is permitted through secure login credentials to your local network VPN.
Configuration of the authorized user’s account credentials are based on the scope of their access, where the scope is defined by their 3-digit MIS code. In this context, the scope of their access is defined as <misScope>. For example, authorized district users will have a scope of access that is represented by their district MIS code (example: 210). College Authorized college users will have a scope represented by their college MIS code (example: 211).
| Info | ||||||
|---|---|---|---|---|---|---|
| Panel | ||||||
| ||||||
The <misScope> will appear in the user’s credentials (username) and also in the database schema formats used in SQL queries during the Connecting Using Connect using SQL Workbench validations. |
...
| panelIconId | atlassian-check_mark |
|---|---|
| panelIcon | :check_mark: |
| bgColor | #F4F5F7 |
...
Connect to the Data Warehouse (Redshift Cluster)
This section is designed for authorized colleges looking who want to gain access to the CC CCC Data Warehouse Redshift cluster databases. AWS Redshift is a secure, cloud-based data warehouse service used for collecting and storing large-scale data sets and enables users to analyze data using various BI tools.
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Note: Colleges achieve direct connection access to the CCC Data Warehouse using SQL Workbench. |
Connection Requirements
In order to complete the connection process (SQL Workbench install) and access the CCC Data Warehouse (Redshift cluster), you the college/district will need to meet the requirements below.
Complete VPN Configuration & Log In from Internal Network: Log in and connection to the Redshift cluster must originate from your local network.
Account Login Credentials: Obtain your account credentials (username and password) and the database endpoint (DNS), provided by your ES implementation team.
Database Connection Strings & Schemas: Ensure the database names, connection strings, and schema roles are correct to access specific databases (see Table B: Application Database Names below).)
Remote Connection: If you are working remotely and the district has authorized it, you will need to establish a connection to the district’s internal network using a separate VPN client that the district has provided for this purpose.
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| Next Step: Follow the instructions to install SQL Workbench and configure database access connection.||||||
| Note | ||||||
Notice: Beloware instructions for implementing SQL Workbench to connect to the CCC Data Warehouse. SQL Workbench is IMPORTANT: Beloware instructions for implementing SQL Workbench to connect to the CCC Data Warehouse. SQL Workbench is supported by the CCC Technology Center, and is recommended to assist with initial data validation but is not a requirement. |
Connect
...
using SQL Workbench
Follow the instructions in this section to install SQL Workbench and configure database access connection. SQL Workbench is a database GUI used for accessing many different databases. The instructions below are similar to those found on the AWS website for SQL Workbench, but are more in-depth regarding the actual specific installation steps.
Process OverviewThe process for connecting to the Data Warehouse using SQL Workbench is as follows:
Prerequisite: Install Java (11 v11 or higher recommended)
| Info | ||||||
|---|---|---|---|---|---|---|
| Panel | ||||||
| ||||||
For more information, visit the AWS website for Connecting to Your Cluster Using SQL Workbench. |
Prerequisite: Install Java
SQL Workbench requires a Java 8 version 11 (or higher) runtime environment. You can use either use a JRE ("Runtime") or a JDK ("Development Kit").
Build 126 is the last version to support Java 8, following versions will require Java 11 or higher.
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
For more information, see Section 4.1: Installing & and Starting SQL Workbench of in the SQL Workbench Manual for additional information. | ||||||
| Note | ||||||
Strongly Recommended: SQL Workbench/J requires Java 8 or later; however, Java 11 or later is highly recommended, especially with High-DPI screens. |
Download & Install SQL Workbench
Download SQL Workbench/J here: https://www.sql-workbench.eu/downloads.html.
Select the generic installation package for all systems with the optional libraries. (We recommend saving this somewhere the package on your local machine with an informative name that is easy to access on your local machine.
...
.)
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Note: The generic package contains the jar file, the manual (HTML and PDF), shell scripts for Linux/Unix based systems (including MacOS) to start the application as well as a Windows® launcher and sample XSLT scripts. |
...
Unzip the downloaded folder and select the sqlworkbench.jar (JAR) file inside to open SQL Workbench.
...
| Note | ||||||
|---|---|---|---|---|---|---|
| Panel | ||||||
| ||||||
IMPORTANT: Follow these instructions to generate an SQL Workbench executable file: |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
More details about installing and configuring the application can be found in the SQL Workbench manual | ||||||
| Note | ||||||
You will need . |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
IMPORTANT: It is necessary to configure the necessary JDBC driver(s) for your database before you can connect to a database. Please refer to the chapter JDBC Drivers for details in the SQL Workbench manual for details on how to make the JDBC driver available to SQL Workbench/J. |
The application is now installed and the Select Connection Profile screen appears.
...
...
| panelIconId | atlassian-check_mark |
|---|---|
| panelIcon | :check_mark: |
| bgColor | #F4F5F7 |
...
Download Redshift Drivers and Test Connection
You will need to download Download the Amazon Redshift JDBC driver version 2.0 to enable SQL Workbench to communicate with Redshift. The file you need Below is the zip file to download is below. Zip File: :
Once you have downloaded and unzipped the Redshift drivers to a directory of your choice you choose on your machine, go back to your the SQL Workbench installinstalled and select `Amazon Redshift` 'Amazon Redshift' from the Driver dropdown.
You should see a At the prompt asking you to configure the driver. Select ‘Yes’, select ‘Yes,’ as shown below.
...
The ‘Manage Drivers’ Manage Drivers menu will come upappear. Click the Folder icon and navigate to the directory that contains the driver package that you recently unzipped, as shown below.
...
Select the Redshift driver, which will be named like ‘Redshift*.jar’ jar,’ then click ‘Open’‘Open.’ See the example in the screenshot below.
...
Once your the driver file is selected, click ‘OK’ on the Manage driversDrivers screen.Panel
| panelIconId | atlassian-check_mark |
|---|---|
| panelIcon | :check_mark: |
| bgColor | #F4F5F7 |
Configure the Database Connection String
Configure the Connection Profile screen with your account credentials (username and password) and the connection string.
...
With the drive selected, a sample endpoint will autofill in the URL field that will need . This endpoint needs to be modified, and several other fields will need to be filled outcompleted. See the example below.
...
The important essential fields to configure are as follows:
Name: The name for this the connector that you will reference later when you want to access this database again. This is for your use only, so make it descriptive enough to where you understand clearly which DB you are accessing. clearly descriptive for the database you will be accessing. Identifying the environment in the name is highly recommended. (For example, if you are going to query the COCI DB, something like a name such as `DWH-COCI` might be appropriate. Signifying the environment in this name is highly recommended.)
URL: Where to reach the database. Edit the autofilled string here with your auto-filled string in this field with the custom endpoint (example shown below).
Environment | Endpoint |
|---|---|
Production | jdbc:redshift://dwh-prod.ccctechcenter.org:5439/{{database}} Ex: jdbc:redshift://dwh-prod.ccctechcenter.org:5439/canvas2 |
|
Where “database” is the name of the
|
Username: the The PostgreSQL username you are using to login.
Password: the The password for the username above.
Place a checkmark in the Autocommit box (, as shown in the example below.)
...
Click `Test` Test in the lower right to verify you have connected successfully. Then click OK.
...
Table B: Application Database Names and Schema Formats
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
The value of <misScope> can be derived from the prefix of your Data Warehouse Direct user name which is in the format of <misScope>_<firstInitial><lastName> i.e. 000_jdoe, where 000 would represent their represents the MIS scope value. An example of the resulting schema for accessing tables within the Application database = dw_apply_read_000. |
Application (Data Source) | Database Name | Query Schema Format |
|---|---|---|
CCCApply Application | application | dw_apply_read_<misScope> |
CC College Promise Grant | bogfw_application | dw_apply_read_<misScope> |
CCC International Application | intl_application | dw_apply_read_<misScope> |
Multiple Measures Placement Service (MMPS) | mmps | dw_apply_read_<misScope> |
Chancellor's Office Curriculum Inventory (COCI) | coci | dw_coci_read_<misScope> |
Course Identification Numbering System (C-ID) | cid | dw_cid_read_<misScope> |
Canvas Data 2 | canvas2 | dw_canvas2_ |
read_<misScope> |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Note: Data dictionaries for the above data sources are available on the CCC Data Warehouse documentation site. | ||||||
| Note | ||||||
Troubleshooting: If the |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
IMPORTANT: For troubleshooting if the connection test fails, the two most common fixes are as follows:
|
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Next Step: Validate your connection and data access. |
Data Access Validation & Testing
...
Data Access Validation & Testing
Once connected to the CCC Data Warehouse with your own tool, colleges are encouraged to conduct a series of data access tests and activities, which may include:
...
If a local data warehouse is available, configure a CCC Data Warehouse table as an “external table” external table in the local database source. This supports data within the CCC Data Warehouse to be connected to your local data warehouse without the need to copy data to the local machine or district server.
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Note: CCCTC College Staff Support is available to assist with issues connecting to the CCC Data Warehouse. |
ES works will work with college colleges to validate initial data queries.
Using provided user documentation (such as this user guide), the college configures their BI tool to connect to the Data Warehouse.
College will confirm The college confirms local connection and successful login, as well as perform and also performs optional data validation activities.
Online documentation is available to support data access validation activities for Researchers.
...
| panelIconId | atlassian-check_mark |
|---|---|
| panelIcon | :check_mark: |
| bgColor | #F4F5F7 |
...
researchers.
Work with your ES ICE (engineer) to confirm access validation and query tests. Discuss any unfinished implementation steps before hand-off to ES Support.
...
Canvas Data 2 with DW Direct Connect Service
Colleges may request to access their Canvas Data 2 (CD2) data through the CCC Data Warehouse via direct access connection (ODBC/JDBC). The ability to access CD2 data will require the college Canvas Administrator to generate and pass their Canvas Data 2 API credentials to the CCCTC Enabling Services (ES) Implementation Configuration Engineer (ICE) as part of the configuration process. Once received and implemented, the Canvas v2 data pipeline warehouse integration will be configured to pass your CD2 data to the CCC Data Warehouse. Following that initial pass, the data will be updated nightlyper a configured schedule.
To get started, please send an email requesting Canvas Data 2 Data Warehouse Direct Connect access for your college to your College Experience Manager (CEM) at CCCTC Enabling Services.
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Important: Colleges must be live with the basic DW Direct Connect service in order to implement Canvas Data 2 Direct Connect access. If your college has not yet configured site-to-site VPN access with the CCC Data Warehouse, please contact Enabling Services to get started. |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Learn For more information about the Canvas Data 2 Direct Connect service, including the college preparatory requirements before and during the implementation process: /wiki/spaces/ES/pages/3172564993. |
Making Queries
...
, refer to the following link: Accessing Canvas Data 2 in the Data Warehouse. |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
For information about the latest Canvas Data 2 updates to the CCC Data Warehouse, refer to the following link: Canvas Data 2 Changes to Data Warehouse. |
...
Access In-Progress Application Data in the CCC Data Warehouse
The CCCApply “in_progress_applications” table is available in the Data Warehouse via Direct Connect Access to make it easier for researchers to perform quick queries without having to include the joins manually for every query.
See related content in Accessing In-Progress Application Data in the CCC Data Warehouse, featuring tables and query information.
...
Making Queries
After the database connection is validated, you will be given the following prompt that will allow you to entry enter SQL Queries.
...
A tab labeled Database Explorer displays more information about the database that your user has access to. Table names can be found herethere, as well as some additional basic additional information about the DB.
...
Finally, you You can also run SQL against objects your DB user has access to.
...
Example Query to run on Data Warehouse Data
To get a row count of a table you can , run the following query:
| Code Block | ||
|---|---|---|
| ||
select count(*) from dw_apply_read_523.contact; |
...
A: While the login credentials are intended to serve as a Service Account for the connection of local applications to for the connection of local applications to the CCC Data Warehouse, individual accounts may be requested for authorized users where this is needed.
More Information: See CCC Data Warehouse - Frequently Asked Questions for more frequently asked questions.
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
To Get Help and Share Info: After successful implementation, if you need help using the CCC Data Warehouse, |
...
| Info |
|---|
More: See the CCC Data Warehouse Frequently Asked Questions for more information and frequently asked questionsaccess the CCCTechnology.info support channel. In that forum, you can get help from the ES Support team. In addition, you can discuss use of the CCC Data Warehouse with other colleges and districts. |