Project

General

Profile

Documentation » Documentation 2022-1 »

Engine For Management of Adaptive Pedagogy (eMAP)

Back to Table of Contents

Description

This text outlines the potential course flow within GIFT managed by the Engine for Management of Adaptive Pedagogy (eMAP). The use case described involves a single ‘lesson’ comprised of four concepts. The concepts are distributed among two Merrill’s Branches within a course configuration. Below will be a set of production rules (if-then statements) that dictate course flow and the dependency with learner model attributes as an individual progresses through the lesson.

Learner State Attributes

Current Attributes:

Cognitive Knowledge (Novice; Journeyman; Expert)

Affective State (Engagement; Frustration; Confusion; Boredom)/(Short-term; Long-term; Predicted)

Cognitive Skill (Novice; Journeyman; Expert)

Used to differentiate ‘knowledge’ from ability to execute. This falls in line with the notion of Knowledge/Skills/Abilities (KSAs) defined in most doctrine and helps to make competency badging within a domain more granular

Behavior State (Notional)

Example Course Flow

  • Learner Login
    • Upon login, learner states are set at Unknown
    • Learner History update Learner State (to be developed)
      • If data is available on the learner (LMS/LRS), then update knowledge and skill attributes
      • If no data is available, then set knowledge and skill attributes to ‘Novice’
  • Select Course
    • If LRS data exists in line with Aptima's Interoperable Performance Assessment (IPA) emerging standard and the Advanced Distributed Learning eXperience Application Programming Interface (xAPI) standard, then take one of recommended courses
    • If lesson is part of course, then open assigned package
    • If lesson is part of self-regulated training, then allow learner to select any lesson available on the machine/server

The following is an overview of a course with these course elements authored:

  1. Pre-lesson assessment
  2. Contextualize Lesson/Gain Attention/Activate Prior Knowledge (Used as course transition element)
  3. Merrill’s branch point
  4. Contextualize Lesson/Gain Attention/Activate Prior Knowledge (Used as course transition element)
  5. Merrill’s branch point
  6. Post Merrill’s Branching Knowledge and Skill Assessments
  7. AAR
  • Pre-lesson Learner Profile Update
    • If the lesson acts on learner relevant data (e.g. motivation, grit, etc.) then it should present survey(s) to the learner (authored through the Survey Authoring System (SAS) ) to update those attribute values
      • If the data already exists for persistent variables in the learner’s long-term model, then update and bypass survey delivery (ie. “skip the pre-test”).
  • Pre-lesson Assessment (survey delivered)
    • If lesson pre-knowledge assessment exists, then deliver through assigned survey context and key in the Survey Authoring System (SAS)
      • If pre-established scoring conditions exist (GIFT’s default), then update learner model based on assessment outcomes
        • Only cognitive knowledge is updated based on performance outcomes within a survey delivered assessment
          • However, the team can see this being an issue within heavily cognitive domains like your traditional tutors of algebra and physics. How to handle heavily cognitive domains, where there is little applied knowledge or practice, is an open area of research.
    • If lesson pre-skill assessment exists, then deliver through assigned training environment, gateway, and Domain Knowledge File (DKF)
      • If pre-established scoring conditions exist (GIFT’s default), then update learner model based on assessment outcomes
        • Only cognitive skill is updated based on performance outcomes within a training environment delivered assessment
          • I’m sure there will be some unforeseen issues with this assumption as well, but it passes the initial eye check
  • Contextualize Lesson (Quadrant?)/Gain Attention/Activate Prior Knowledge

This is an optional course element/transition that aligns with Gagne’s nine instructional events. It can serve as a node in the course file that refers to some pre-authored material to serve this purpose.

  • Enter Merrill Quadrant 1 (Concept A: Read Map and Plot Points)
The algorithm for selecting the appropriate content for the Rule and Example quadrants can be found in Metadata/Content Selection Algorithm.
  • Rules: Configure material around defined concepts being instructed and known attributes of the learner that match entries within the eMAP’s decision tree
    • Attributes
      • Cognitive Knowledge; Motivation; Self-Regulatory Ability; Self-Efficacy
    • Proposed Assessments
      • Affective State: monitor learner to assess emotional and cognitive reaction to material
      • Behavior: monitor behavior within learning environment to assess gaming behaviors
    • No knowledge/skill updates within learner model
  • Examples: Configure material around defined concepts being instructed and known attributes of the learner that match entries within the eMAP’s decision tree
    • Attributes
      • Cognitive Knowledge; Motivation; Self-Regulatory Ability; Self-Efficacy
    • Proposed Assessments
      • Affective State: monitor learner to assess emotional and cognitive reaction to material
      • Behavior: monitor behavior within learning environment to assess gaming behaviors
    • No knowledge/skill updates within learner model
  • Recall (Knowledge Assessment)
    • If a bank of questions for this concept has been authored within the SAS, then deliver randomized recall assessment based on eMAP configuration (configuration is defined within GIFT’s Course Authoring Tool)
      • If pre-established scoring conditions exist (GIFT’s default), then update learner model based on assessment outcomes
  • Assumption: Only cognitive knowledge is updated based on performance outcomes within a survey delivered assessment within the recall quadrant
    • Guidance Configuration (to be developed)
      • Use known attributes of the learner to configure timing and specificity dimensions
  • Question by Question Feedback vs. Following All Items
    • Attributes that may dictate this decision: Knowledge and Self-Regulatory Ability
      o General to Specific vs. Specific to General Feedback
    • Attributes that may dictate this decision: Knowledge and Self-Efficacy
    • Remediation
      • If learner is reported at ‘below expectation’/’at expectation’ on any items (i.e. concepts), then initial remediation loop within the defined Merrill Branch
  • Remediation path is dependent on reported cognitive knowledge state based on defined scoring logic in the Course Authoring Tool
    • For each concept:
      • If learner is scored at ‘below expectation’ based on scoring configuration, select that concept for Rule quadrant remediation
      • If learner is scored at ‘at expectation’ based on scoring configuration, select that concept for Example quadrant remediation (can be in addition to Rule quadrant remediation)
    • If there is any concept remediation needed, present the Rule remediation for all identified concepts (above) first followed by Example remediation.
      • This is where the metadata selection algorithm is used to select different content to deliver to the learner (if available).
      • Remediation ends back in Recall Quadrant
  • If items report at ‘below expectation’ again and there is no new content to present; then allow the learner to select the quadrant they prefer to remediate in (to be developed).
    • If all items in the Recall Assessment are reported at ‘above-expectation’ then move onto Practice.
    • If no questions exist for this concept within the SAS or the author removed the recall quadrant from this branch, then move onto Practice.
  • Practice (Skill Assessment)
    • If no practice has been authored/configured, and the Recall Quadrant has been satisfied, then move onto next transition in the course file
    • If a training environment/scenario has been configured, then deliver practice materials through pre-established Gateway and DKF
    • Configure material around known attributes of the learner that match entries within the eMAP’s decision tree (to be developed)
      • Attributes
  • Cognitive Skill; Motivation; Self-Regulatory Ability; Self-Efficacy
    • Proposed Assessments
  • Affective State: monitor learner to assess emotional and cognitive reaction to material
  • Behavior: monitor behavior within learning environment to assess gaming behaviors
  • Skill: monitor performance in real-time across all identified sub-concepts based on pre-defined assessments authored around Evidence Centered Design (Stealth Assessment)
    • If pre-established scoring conditions exist, then update learner model based on assessment outcomes
    • Assumption: Only cognitive skill is updated based on performance outcomes within a practice environment
    • A survey authored in the SAS can also be defined as a practice environment (to be developed).
    • Guidance Configuration (to be developed)
      • Use known attributes of the learner to configure timing and specificity dimensions
  • Number of violations before triggering guidance/feedback
    • Attributes that may dictate this decision: Skill and Self-Regulatory Ability
  • General to Specific vs. Specific to General Feedback
    • Attributes that may dictate this decision: Skill and Self-Efficacy
  • Static (text or audio alone) vs. interactive (AutoTutor reflection)
    • Just an idea here…needs fleshing out
    • Remediation
      • If learner is reported at ‘below expectation’/’at expectation’ on any items, then initiate remediation loop within the defined Merrill Branch
  • Remediation path is dependent on a combination of skill and knowledge
    • If learner is novice in skill and expert in knowledge, then re-initialize practice
    • If learner is novice in skill and journeyman in knowledge, then navigate to examples quadrant
      • (Notional) A failure to apply knowledge results in the presentation of examples of the knowledge applied.
      • Remediation ends back in Recall Quadrant (to be developed)
  • If items report at ‘below expectation’ again and there is no new content to present, then allow the learner to select the quadrant they prefer to remediate in
    • If all items in the Practice Assessment are reported at ‘above-expectation’ then move onto next transition in the course file
  • Contextualize Next Quadrant/Gain Attention/Activate Prior Knowledge

This is an optional course element/transition that aligns with Gagne’s nine instructional events. Can serve as a node in the course file that refers to some pre-authored material to serve this purpose

  • Enter Merrill Quadrant 2 (Concept B & C: Route Planning and Calculate Distance/Azimuth)
  • Rules; Examples; Recall; and Practice as Described Above for new defined concepts
  • Assumptions
    • New concepts are dependent on requisite knowledge/skill delivered and assessed in previous quadrant
    • Based on Mastery Learner, it is assumed that a learner has enough understanding of prerequisites for progression into this Merrill Branch
    • Rules and Examples remain void of assessments that dictate knowledge and skill attribute updates
    • Recall and Practice can potentially include assessments linked with concepts covered in previous branch (to be developed)
      • Recall assessments delivered will include items from previously instructed concepts to ensure retention.
  • Post Merrill’s Branching Knowledge and Skill Assessments (to be developed)
  • Option to deliver a knowledge or skill assessment using the SAS or a training environment
    • Intended to be void of guidance functions to determine how the learner performs on their own
    • Can be used to trigger more remediation loops or to establish final score and attribute values for a lesson
  • After Action Review
  • Used for reflection and summarization practices
    • Reflect on the experience and the assessment outcomes
    • Summarize the objectives of the lesson and why it is important
      • Both can be static or interactive based on the defined approach
      • Information on performance along with a summary narrative can be provided or an AutoTutor interaction can be initialized
  • There are a number of options that can be utilized here
  • Lesson Completion
  • Update LMS/LRS with outcomes values of knowledge and skill attributes for all concepts and sub-concepts scored
    • This can be in “xAPI” format to support IPA practices

Additional Examples

SCENARIO 1A
  1. Single concept adaptive courseflow course object
  2. No / Novice prior knowledge

SCENARIO 1B
  1. Single concept adaptive courseflow course object
  2. No / Novice prior knowledge
  3. Optional Practice

SCENARIO 1C
  1. Single concept adaptive courseflow course object
  2. Journeyman prior knowledge

SCENARIO 1D
  1. Single concept adaptive courseflow course object
  2. Expert prior / Novice knowledge

SCENARIO 2A
  1. Multiple concept adaptive courseflow course object
  2. No / Novice prior knowledge

SCENARIO 2B [Same as 2A]
  1. Multiple concept adaptive courseflow course object
  2. Journeyman / Expert prior knowledge Concept A, No / Novice prior Knowledge Concept B
    or
    No / Novice prior Knowledge Concept A, Journeyman / Expert prior knowledge Concept B

SCENARIO 3
  1. Single concept adaptive courseflow course object
  2. Practice covering concepts taught in this course object (Concept C) as well as previous course objects (Concepts A & B)

Metadata/Content Selection Algorithm

This section describes the current selection algorithm when using Merrill’s branch point course element in a GIFT course. The logic only applies to the Rule and Example quadrants for each Merrill’s branch point course element.

Prioritization Logic:

1. If possible, don’t present domain content that has already been presented.
2. Maximize the needed coverage of concepts by selecting the fewest content to present at that time.
-- i.e. don't waste the learner's time, allow them to receive a quick refresher on important information and then return back to the assessment activity.
3. Maximize the appropriateness of the content by selecting the best match of eMAP learner state attributes to the available metadata attributes.
-- The system might not have all of the Cognitive, Affective and biographical information about the learner, so make the best of what's available.
4. If available, use the content’s paradata to trim the number of content choices that cover the same set of needed concepts.
-- Paradata describes how the content has been used in the past. This information can help further down select the content to present.
5. Choose randomly from the content choices that cover the same set of needed concepts.