Cogniti

1. Ensure that your default email in Quercus is “utoronto.ca” or “mail.utoronto.ca” (It can not be rotman.utoronto.ca, dentistry.utoronto.ca, etc.).  See this guide on how to change your default Quercus email.
2. Manually add the Congiti UAT link to your Quercus course menu. (Instructions below “Access Cogniti from your Quercus course“)
3. Select/click on the Cogniti UAT link one time.  This will create your account in the Cogniti system, however you will not be able to create agents yet.
4. Request Ability to Create Cogniti Agents. Wait for confirmation that your account has been approved to create Cogniti agents.

Add the Cogniti UAT access link to your Quercus course:

1. In your Quercus course, select “Settings” from your course menu.
2. Select the “Navigation” tab.
3. Look for “Cogniti UAT“.  Drag it up from the ‘hidden from students’ area and drop into the visible section of your course navigation list.
4. Scroll down and select the blue “Save” button.

The “Cogniti UAT” link should now be listed in your course menu:

(alternative to the drag & drop, you can select the vertical ellipsis / 3 dots and choose “+ Enable”, then Save):

.

Instructors (& TAs) must always use the “Cogniti UAT” Quercus course menu link when creating and editing Cogniti agents.

• Other people helping you build/edit your agents like TAs or additional support staff, access is with this same link in your Quercus course menu.
• Students will NOT see the Cogniti UAT button in your course menu, only Teacher and TA roles in your Quercus course.

The Cogniti UAT link will open up a new browser window with the full Cogniti interface at a cognitiproject URL.  DO NOT save or bookmark this cognitiproject URL, as you can not access Cogniti alone via this URL.

If you are trying to access the main Cogniti interface and receive an error message, close your tab/window and try again through the Cogniti UAT link in your Quercus course.

The Cogniti main menu looks something like this where you can choose from the following options:

• “Create agent” (you will not have the ability to create agents unless you have been approved for the Virtual Tutor pilot)

Once you have created your first agent, you will see the following:

• Start a chat with your agent (to test its abilities)
• Edit your agent.  The 2 main areas you will want to edit are:
  • Prompts.  This is where you customize your system message – how your agent behaves and defining its purpose.
  • Resources.  This is where you add course files which can be used by your agent when providing responses.
• Insights provides an auto generated summary of what kinds of topics and questions your students have been asking
• Conversation history is a log off all the chat transcripts that have occurred with your agent (student names are removed).

Students will NOT see the Cogniti UAT link in your Quercus course menu.

• Students can only access the specific Cogniti agents that you choose to add to a Quercus course page.

Cogniti agents are added to your Quercus course via the tools in the Quercus Rich Text Editor.  This is how students will access your Cogniti Agent.

1. Create a new Quercus Page.  We recommend making the Quercus page Title the same as your Cogniti Agent Name (to keep track of which Quercus page is associated with which Cogniti Agent).

2. Edit the Quercus page, select the icon that looks like an electrical plug, and select “Cogniti UAT”:

• The electrical plug ‘apps’ icon will be beside the red OneDrive icon in the RCE toolbar.  If you do not see this icon, select the vertical ellipses (3 dots).
• You might need to select “View All” from the electrical plug ‘apps’ icon first and then select “Cogniti UAT” from the list

3. Browse the listing of agents you have created or have access to, and choose “Select this agent” to add it to your page.

Everyone listed in the ‘people’ section of your Quercus course (students, TAs, etc.) can chat with this specific Cogniti Agent via this Quercus Page. 

You can also share this Quercus Page URL (not the Cogniti URL) with your students in other areas like Announcements, email, using the Quercus redirect tool, or in other approved teaching and learning tools like Piazza.

• Encourage your student to provide feedback to Cogniti when they interact with the chatbot.  They can rate by hovering over the Cogniti response and selecting a Thumbs-up or Thumbs-down.  Inappropriate responses by the chatbot can be Flagged.  Students can include written comments when providing response feedback.
  

• Reinforce with your students that they should refrain from including personally-identifying information in their chats.
• Student identities are not included in chat history – instructors will have access to de-identified data only.
  
• Students can manually export their own person chat history: How can I see my past conversations with agents? – Cogniti

A Cogniti Terms of Use acknowledgement is presented to students upon first use of the chat tool, and can be accessed anytime at the bottom of every Cogniti chat window along with a reminder about not sharing private information and data collection: 

Adding course resources is done by editing your agent (the pencil icon), and the choosing the “Resources” section.

This is what the main “Resources” interface looks like (Canvas resource = Quercus):

Do you want the agent to link directly to the actual reference file that students can open, or just find the answer and indicate the answer was guided by the added references?  If you choose to enable the option “Agent should cite any resource(s) that it uses“,  the referencing capability depends on how materials are uploaded in the Resources area of your agent:

• Web page → the AI agent can both use the info and provide a reference (hyperlink) back to the whole source web page.
• File (uploaded directly in Cogniti) → the AI agent can read/search inside it and use the info, but it does not provide a link back to the file itself.
• Canvas resource (uploaded in Quercus and then attached to Cogniti) → the AI agent can both use the info and provide a reference (hyperlink) back to the whole source document.

In the Cogniti system, “Canvas resource” means adding a file that is already in your Quercus course.  Upload files first to your Quercus course and then select them from Cogniti.

Notes about adding resource files:

• Resources can only be text searchable (no imbedded images or videos will be included).
  • You can however include text transcripts of videos (lecture video transcripts for example).
  • If there are images in your files that contain significance, provide a text description or surrounding explanations.
  • Powerpoint files should be saved/exported as text readable pdf files.
  • Only the text part of files will be readable to the agent.  Your agent will not know context about images, videos or any other other non-text elements.  That information will need to be created, so your agent has all of the information it needs to provide good responses.
• You can add files in a batch, but you can also only update/remove them in that same batch.  We recommend adding them in smaller groups or one at a time to allow for flexibly in later changes.
• For library resources we recommend adding the pdf from your Quercus course (rather than a public or library website URL).
  • While most library resources copyright is covered the same way in Cogniti as it is to upload the file into your Quercus course (i.e. only your registered students can access the file), it is still a good idea to check with your Faculty Liaison Librarian.
• If you want your agent to cite (link to the source file) in your Quercus course, remember that file must be available/published in your Quercus course for students to access.
  • You can choose any file from Quercus (when selecting, you will see either a check mark or X).  This indicates if the file is currently available/published for students in your Quercus course.

A few notes about how agents use your resources:

• In your system message, do not refer to your resources – this will not make sense to the underlying AI, because all it sees are little snippets, not “the document / file”
• Only snippets of the resources are sent to the large language model for consideration, not the full resource.
  • “kind of” like when you make a standard google search – only keywords/phrases are found and extracted for use by the model when it formulates it’s response.
• If you have information that is critical to the function of your agent, ensure you put that information into the system message.
• Best to chunk into focused areas, rather than including a lot of full text that may or may not be as relevant when the agent looks for answers.
• It is also a good idea to “clean” your resources for better quality and more accurate responses from your agent. Please visit our CTSI guide on Preparing Content for Custom AI Chatbots: Guidance on how to organize, clean, and format course materials for use with custom AI chatbots, ensuring accurate, relevant, and efficient AI responses:
  Web page | PDF | Word

Access your system message by editing your agent (the pencil icon), and the choosing the “Prompts” section:

Consult the following resources to assist with building your custom System message (system prompt):

CTSI Teaching with Generative AI resources – 

• Effective Prompting Strategies: Offers evidence-based prompting techniques and sample questions to help students get the most out of AI virtual tutors, supporting personalized learning, critical thinking, and deeper understanding of course material.
  Web page | PDF | Word
• Developing an Effective System Prompt: Offers strategies to translate your teaching goals into effective system prompts that scaffold student learning and maintain pedagogical integrity.
  Web page | PDF | Word

Teaching @ Sydney resources:

• How to design custom AIs that can propel learning – The role of system message.
• Cogniti guides – How do I design a good system message?

You can give access to other people helping design/build/edit your agent. This requires three steps:

1. Add them to your Quercus course.

In the “People” section of your course, you can add someone with their UTORid username, or email address.

Their role must be of “Teacher” or “TA”.

2. Once added in Quercus, that person must select the “Cogniti UAT” link in your Quercus course menu.
(By them entering the Cogniti system for the first time, their account in Cogniti will be created.)

3. In the “Sharing” area in your Cogniti Agent (select the pencil ‘edit’ icon, and then “Sharing”), search for and add the person in the “Who else can edit and administer this agent?” section.

What is a sandbox?

• A sandbox is an educator-controlled space where students can create their own AI agents.

The Sandbox feature is not enabled by default. 

• Please contact Derek in MS Teams or the CTSI GenAI in Teaching & Learning Teams channel for information and access for this Sandbox feature.

Once you have been given access to create sandboxes, look for the Sand Pail icon in your Cogniti menu:

#1 – Create a Sandbox space for your students.  This sandbox is a space/folder where your student will create their agents within.  Think of it as a little mini-Cogniti space of their own in your course

The following settings must be set/checked before you can create sandboxes for your students:

• Details: required.
  • You must give your Sandbox a name.  To avoid confusion, create a name that includes your course name and is related to the activity.
• Control Prompt: optional.
  • A sort-of prompt checker. The instructor can put some rules in place for what students can enter as their student-created system prompt.
• Sandbox access: required.
  • Change to the default “anyone who accesses this sandbox via an LTI connection“.
  • You must specify a sandbox access period (including both a date and time).
• Agent restrictions: required.
  • Broadest access level of agents in this sandbox: This option allows students the choice to share their agent with the rest of the class – change to “Anyone who can access this sandbox”.
  • Agent Quota per student: Must change from 0 (zero) to at least 1.
• AI models: required.
  • Must choose from one of the allowed AI models
• Ownership: optional.
  • leave as default, unless you have course staff assisting with the development and administration of your sandboxes.

#2 – Give students a link to create an Agent in the Sandbox.  Students access to the Sandbox is almost the same as a regular cogniti agent, only this time you select from the “Sandboxes” tab, rather than from your list of Agents:

#3 – Other notes about Cogniti Sandboxes:

• As the instructor, you will have access to see all of your student’s agents and access all agent information:
  • Start Chat, Edit Agent (to view their system message, settings), View Insights, View Conversation History.
  • You must change your view from Card View to List View to see your student’s agent information:
• However, the list will be displayed using the Agent name created by your students.  You can not view/sort them by student name or any other attribute, only agent name. A good practice might be to ask your students to name their agents using their own name as a way for you to view all of their agents.
• By default, students can only see and interact with their own agents.
  • Students can adjust this to share their Agent with the class.
  • In the student’s Agent settings -> Chat access -> Who can find this Agent? / Who is authorised to chat with this agent? – Change to “Anyone with access to this Sandbox”.

Once your instructor has setup the Cogniti sandbox area, the quercus page should look something like this:

Creating your Cogniti agent:

1. Select the blue “(+) Create agent” button near the top right corner of the Cogniti sandbox
2. Select “Start with a blank agent”
3. Give your agent a “Name“ and short description.  (Follow the instructions given to you by your instructor in terms of the proper naming convention they would like you to use.  This is important for your instructor to identify your agent.)
4. You can enter a “System message“, however this can always be added/modified later.
5. In general, do not modify the Temperature or AI Model unless your instructor give guidance for this.
6. Save agent

Chat or edit your Cogniti agent:

Once you have created your Cogniti agent, you can chat with your agent using the chat bubble icon, or edit your agent settings using the pencil icon:

Share your agent with your classmates:

By default your agent is private to you and your instructor.  If your instructor allows, you can choose to share your agent with the class.  Change these setting (in the pencil / edit agent area):

1. Edit your agent (edit area / pencil icon)
2. Select the “Chat access” section
3. In the section “Who can find this agent?“, change to “Anyone who can access this sandbox”
4. In the section “Who is authorized to chat with this agent?“, change to “Anyone who can access this sandbox”
5. Save agent

(note: do not use the Secret link URL option – it does not work and will cause errors)

1. Start by downloading your anonymized Cogniti student chat data (the clock symbol for your Cogniti agent):
   
2. Then select “Export all conversations” to download a CSV file to your computer or OneDrive.
3. Open Microsoft Copilot (or another University approved tool), attach your CSV transcript file.
4. Begin with a broad orientation prompt such as: “I have uploaded anonymized student chat transcripts from an education-focused generative AI chatbot.”.  Add onto your prompt with some targeted suggestions below:

Course Alignment.
Were students using the tool in ways that matched your learning objectives? Look for off-topic use, surface-level requests, or gaps in expected content areas.
Copilot Prompt ideas:

• “Summarize the main topics students asked about. How well do these align with a course focused on [course topic]?”
• “Were there any major course topics that appeared rarely or not at all in student conversations?”

Common Points of Confusion.
Recurring questions or repeated rephrasing of the same question are strong signals that a concept needs more attention in instruction.
Copilot Prompt ideas:

• “What topics or questions came up most frequently across all student conversations?”
• “Identify conversations where a student asked follow-up questions on the same topic, which may indicate confusion or difficulty understanding.”

Quality of Student Engagement.
How students phrase their questions reveals the depth of their thinking. Look for evidence of iteration, critical thinking, or shallow one-and-done requests.
Copilot Prompt ideas:

• “Identify conversations where students refined or followed up on their questions. What do these patterns suggest about how students engaged with the tool?”
• “Were students generally asking conceptual questions, or were most requests focused on getting direct answers or completed work?”

Intended Use Patterns.
Are students using the AI as a thinking aid or as a way to bypass the work? Look for patterns that suggest the tool is being used as a shortcut rather than a thinking aid.
Copilot Prompt ideas:

• “Are there conversations where students appear to ask the tool to produce completed work, such as written responses or solutions, rather than seeking guidance or explanation?”
• “Describe the overall pattern of use: are students engaging with the tool to develop their understanding, or does the data suggest they are using it to bypass the learning process?”

Cogniti Response Quality.
Examine whether Cogniti responded in ways that were accurate, helpful, and appropriately scaffolded.
Copilot Prompt ideas:

• “Were there cases where the AI’s response seemed inaccurate, overly simplistic, or gave away an answer rather than guiding the student?”
• “Identify any exchanges where the AI response may have caused confusion or led a student in an unhelpful direction.”

Engagement and Access Patterns.
Variation in how much students used the tool may point to usability issues, uneven awareness, or barriers worth addressing before the next term.
Copilot Prompt ideas:

• “How varied is the length and frequency of student conversations? Were some students much more active than others?”
• “Are there any patterns in the data that might suggest some students had difficulty using the tool effectively?”

Other tips / suggestions:

• Copilot / Generative AI summaries are a starting point.  Always verify patterns by reading transcripts directly.
• Generative AI analysis can miss nuance. Use your professional judgment alongside any generated output.
• Be specific. Name the course, topic, assignment type, or concept you want to focus on.
• Ask Copilot to explain its reasoning: “What in the data led you to that conclusion?”
• If results seem off, try rephrasing: “Look only at student messages, not the AI responses.”

The default settings that come with your Agent are the most desirable.

In general, there are only 2 main areas to edit in Cogniti when you are first getting started:

• Prompts
• Resources

You do not share your agent with students using the URL provided in this Cogniti sharing area – sharing with students is done in Quercus only.

If you wish to share an agent with other course staff for testing, but not students.  You can create your Quercus page as normal, but leave the Quercus page unpublished, so only other Teacher and TA roles in the course can access.

These particular settings should be checked if your students are not able to chat with your agent from within your Quercus course:

Chat Access:

• In the Chat Access settings area, leave the setting under “Who is authorized to chat with this agent” as “Anyone in the organization”
• You control who has access to chat with your Agent via your Quercus course.
  • By adding your agent in your Quercus course, only those with access to your specific Quercus course will have access to chat with your agent.
• This permission does not allow anyone to discover or find your Agent in any other way than through your Quercus course.
  • Only the Owner and Specified Admins (via Sharing) can find the agent.

Sharing:

• Leave the sharing options as “only the owner and specified administrators”
• If you wish to edit (co-author) your agent with other specific people, you can enter their names.  Please note, there is no tracking of changes to an Agent, so if you have multiple people editing the same agent at the same time you will not have any record of who made which changes.