/
Introduction to SuperGlue

Introduction to SuperGlue

Overview

SuperGlue supports the California Community Colleges' many initiatives by providing a technology integration framework for all 116 community colleges in support of more than 2 million students per year.

SuperGlue started in 2016 in response to the growing need to deliver data from systemwide services to colleges using different ERP/SIS systems.

Project SuperGlue combines several technologies:

  • application integration

  • service orchestration

  • API management

  • data integration

These technologies leverage existing open-source technology or cloud services from AWS, as well as developed and managed technologies to support ERP/SIS integrations.

These components are associated with SuperGlue:

Flow of Data

The diagram below illustrates how SuperGlue has integrated many CCCTC enterprise applications with the primary ERPs in use by our colleges.

SuperGlue has also been extended to provide support for writing transactional data into the CCC Data data lake, which is then processed into the CCC Data Warehouse.

The diagram below shows the most common usage of SuperGlue, with the Orchestration Layer feeding to the college’s built-in tables.

Full text description is in the paragraph below.
Diagram 1: Common usage of SuperGlue

Input Sources & Data Points

  • CCC Account: Feeds into CCCID First, Last data fields.

  • CCC Apply: Feeds into misCode, appID data fields.

  • MMPS: Feeds into Placements data fields.

Core Processing

  • CCCTC Orchestration Layer: Represented by a large cloud icon in the center, acting as the hub for data integration.

Integration & Adapters

  • College Adaptor: A box connected to the top of the orchestration layer.

  • Ethos APIs: A box connected to the right of the orchestration layer, serving as an interface for data delivery.

Data Storage (Databases)

  • College Staging Tables: A cylindrical database icon receiving data from the College Adaptor.

  • College Built in Tables: A cylindrical database icon receiving data from both the Staging Tables and the Ethos APIs.

Visual Flow Summary

The diagram illustrates a workflow where multiple application sources (CCCAccount, CCCApply, and MMPS) send specific identifiers and placement data into a central CCCTC Orchestration Layer. This layer then distributes that information through two primary channels—an Adaptor and an API—to eventually populate College Tables.

CVC-OEI Data Integration

SuperGlue has the ability to transmit student application information from the “home” college to the “teaching” college in support of a student registering for an online course. The storage of data in a college ERP is not standardized between different SISs. Furthermore, there are no agreed upon standards even among colleges using the same SIS. 

Flow chart of a student application being stored at the Home College but also being shared with the Teaching Colleges.
Diagram 2: Data Sharing

 

Data could be written into the teaching college ERP as a SQL write, but college concerns over data integrity and governance made that problematic. Instead, treating the transfer as a defined transaction via a web service was preferred. The transfer “across” the wire needs to be in a “standard” or canonical format. As such, utilizing staging tables within each Student Information System provides a standard way of delivery, while then allowing colleges to process each record into the built-in tables as desired to fit individual college needs. 

College Adaptor

SuperGlue for Apply securely delivers CCCApply records - in near real-time - to a staging table in your college IT environment via the SuperGlue College Adaptor. Records include the entire CCCApply dataset, including the college's custom Supplemental Questions. 

Full text description of this image is in the paragraph below.
Diagram 3: College Adaptor

This diagram outlines the communication flow between the central service worker and a college's Student Information System (SIS). The process follows a top-down technical stack within the CCCTC College Adaptor.

Here is the step-by-step flow:

1. External Initiation

  • CCCTC Service Worker: The flow begins here. This component acts as the external orchestrator that triggers requests or exchanges data with the specific college environment.

2. Integration Layer (The Adaptor)

The data enters the CCCTC College Adaptor, which houses three internal layers:

  • Java REST API: This is the entry point of the adaptor. It receives the call from the Service Worker and handles the standard web service communication.

  • CCCTC’s SIS-specific Groovy Code: The data then passes to this logic layer. This is where the general request is translated or "mapped" into logic specifically tailored for a particular college's system requirements.

  • Java Connection Interface: This is the final internal component that manages the actual technical handshake and protocol necessary to talk to the external database or system.

3. External Destination

  • College SIS: The flow concludes with the College SIS, which receives the processed data or request from the Java Connection Interface.

Data Lake and Data Warehouse

As a part of the evolution of SuperGlue, these features have been implemented:

  • application to college

  • determining placement based on multiple measures business rules

  • bi-directional communication of fraud reports between colleges

 

 

The Data Warehouse is a shared space where researchers and the Chancellor’s Office can get data for all colleges.

Diagram 4 shows the state of some of the SuperGlue services. The elements are fully deployed and live supporting all colleges. The College Adaptor is available to all colleges today including schools that use Banner, Colleague, PeopleSoft, and Ethos.

SuperGlue workflow. Shows inputs from Students and Colleges and how it gets transformed and sent to the Data Warehouse.
Diagram 4: SuperGlue Architecture

This diagram illustrates the SuperGlue workflow.

1. Entry & Authentication

  • Users (Students/Colleges): The process begins with users accessing the system through an SSO Proxy.

  • Identity Management: The proxy routes users through OpenCCC Keycloak or SuperGlue Keycloak.

2. Application & Orchestration

  • Application Layer: Users authenticated via OpenCCC interact are directed to the Standard, International, or California College Promise Grant application.

  • Orchestration: Users authenticated via SuperGlue move into the Conductor (API GTWY), which manages traffic to integration services.

3. Integration Services (Service Workers)

The Conductor routes requests to specific Service Workers that act as bridges to external systems:

  • College SIS: Connects via the College Adaptor.

  • Canvas Integration API: Syncs data with Canvas LMS.

  • eTranscript: Manages electronic transcript data.

  • S3 Sync: Moves data into cloud storage for further processing.

4. Data Pipeline

  • Data Lake: Information from the S3 Sync and the CCC Apply modules flows into a central Data Lake.

  • ETL (Extract, Transform, Load): Data from the Data Lake and the application layer undergoes ETL processing to clean and structure the information.

  • Reporting Layer: Traffic from SuperGlue Keycloak is routed through the Reporting Layer (with Business Logic).

5. Final Storage

  • Data Warehouse (AWS Redshift): The final destination for all processed data.

SuperGlue’s impact can be summarized with an example of a potential student who creates an account with OpenCCC for the first time. 

  1. The student explores careers using Career Coach, then completes the application in CCCApply.

  2. SuperGlue kicks off a workflow event.

    Workflow from account creation to triggering a SuperGlue Workflow event.

  3. SuperGlue transfers application data to the associated SIS College Adaptor.

  4. The Adaptor transforms the data to the SIS-specific system.

  5. SuperGlue copies a subset of the application data to the MyPath student profile.

  6. SuperGlue queries California College Guidance Initiative (CCGI) for that student’s high school transcripts.

  7. Profile and transcript data are transmitted to the Multiple Measures Placement Service (MMPS) Placement Engine.

  8. Student placement recommendation is generated, then SuperGlue transmits the placement recommendation to the College Adaptor’s staging table, as well as storing that data in the CCC Data Lake.

Data traveling from SuperGlue Workflow Event through MyPath, SIS staging table and ultimately to the data lake. The steps in the diagram are the same steps as the steps listed above.
Diagram 5: SuperGlue Workflow Event

 

College Adaptor Features

SuperGlue integrations deliver data to colleges via the SuperGlue College Adaptor, which is installed behind the college’s firewall at the college’s request. The SuperGlue College Adaptor provides these features:

  • ability for colleges to receive student application data and verified transcript student placement records directly to their college’s IT environment 

  • support for common workflows and data exchange between systemwide products such as MyPath and the college SIS, and delivers student data to the CCC Data Warehouse

Glue for Apply

SuperGlue’s Glue for Apply feature delivers student application data directly to the College’s SIS via the SuperGlue College Adaptor.

This includes the following CCCApply applications:

  • Standard Application

  • International Application

  • California College Promise Grant Application

  • Non-Credit Application

Glue for Apply delivers the full dataset for each of these applications directly to colleges within seconds of the student submitting them.

For a complete list of data elements delivered for each application, please visit the CCCApply Data Dictionaries page directly or jump to each application section via the links below:

Multiple Measures Placement Service (MMPS)

SuperGlue’s College Adaptor supports the CCCTC MMPS feature by delivering verified student transcript placement information and/or self-reported transcript placement information directly to the college SIS.

Workflow for MMPS Placement Logic showing steps through CCGI, CDE, AB705 Rules Engine and back to MMPS to make a placement recommendation.
Diagram 6: MMPS Placement Logic

 

This flow chart describes the Multiple Measures Placement Service (MMPS) process, specifically how student data is verified and used to generate course recommendations.

Here are the steps in the flow:

1. Data Collection

  • CCCApply Process Steps: The student begins by providing self-reported data.

  • Multiple Measures Placement Step: This data triggers the placement logic.

2. Automated Record Verification (The Decision Tree)

The system attempts to verify the student's self-reported data against official sources:

  • Step A: The system asks, "Does record exist in CCGI?" (California College Guidance Initiative).

    • If Yes, it moves to the data processing stage.

    • If No, it moves to Step B.

  • Step B: The system asks, "Does record exist in CDE?" (California Department of Education).

    • If Yes, it moves to the data processing stage.

    • If No, the system falls back to Self-Reported Data if Needed.

3. Logic & Rules Processing

  • AB705 Rules Engine: All verified or self-reported data is fed into this database/engine. This engine contains the state-mandated logic (AB705) used to determine appropriate course levels.

  • Math/English Placement Recommendations: The engine outputs specific placement levels for the student based on the processed data.

4. Result Delivery

  • Application and Placement Data: The recommendations are bundled with the student’s application.

  • Colleges: The final data and placements are sent to the individual Colleges to be used for enrollment.

For more details on MMPS, see the Multiple Measures Placement Service (MMPS) Placement Logic and Implementation Guide.

Data Lake and Data Warehouse Integration

CCC Data is a combination of data sources stored, governed, and presented in ways which are the most effective for the intended use.  A data lake is deployed to capture data, in its raw form, from operational data stores. 

Multiple Measures (summarized above) also supports the use of Conductor to ensure its data is stored in the data lake as well as in the colleges’ SIS systems. SuperGlue writes MMPS placement data to an S3 bucket using the s3Sync microservice.

Other data integrations, particularly those involving table replication, are not as well suited to SuperGlue. For those, we employ AWS services like the Data Migration Service (DMS) and AWS Glue.

The Data Lake consists of multiple S3 buckets, some of which SuperGlue writes to, some of which do not involve SuperGlue.

SuperGlue is not integrated with the Data Warehouse directly. Instead, SuperGlue passes data to the data lake where it is then processed by an AWS Glue job into the Data Warehouse.