Maddie – The IB cAInine

May 23, 2023

Maddie is an open effort by For.Education, built to further our own understanding of this nascent technology, engage with others who want to explore this potential, and hopefully provide real value to schools.

By their very nature, schools deal with confidential and sensitive information. As an EdTech data company, For.Education puts the security of school data foremost in everything we do.

All of this begs the question, what are schools to do? Is there a way to leverage AI as a sounding board, especially with confidential or highly domain specific information? The new technologies have a lot of potential, but are also third-party, external platforms that aren’t designed with schools in mind.

At For.Education, we are exploring this “problem space” and asking, what place could a generative language model fill in a school that would provide value locally, serving the school and its stakeholders, and no one else.

Your Loyal Companion

Schools are like little villages unto themselves. With divisions and departments regularly home to thousands, and teachers with a multitude of directives on top of what they are already doing – the power to be gained from a personalized chatbot is easy to understand.

Get insights into your school policy and curriculum quickly
Generate communications by leveraging your organizational voice
Semantic search and lookup to index and retrieve school documents quickly
Offer pedagogically/curriculum grounded feedback for teachers during PD and PLC groups

Maddie in Action

Below are some use-cases we would envision for Maddie:

An MYP teacher is preparing an interdisciplinary unit and wants to know what other courses have content that could go well with what they have planned. The teacher can ask Maddie who will be able to perform semantic search on an embeddings database of school content. This is a complicated way of saying that Maddie can fetch units that might blend well with what the teacher is teaching as well as fetching the links and teacher contacts
Maddie can prepare high level summaries of what materials were covered in courses and curricula and prepare class messages reviewing what was covered in course (and also suggestions on how parents can support their children and their learning)
Help teachers to quickly look up school policy and find relevant documentation
Act as an extra voice during PD and PLC groups to help teachers better leverage the existing school resources, and also bounce ideas back and forth

This is a short initial view of some of the ways we hope Maddie might come to serve schools in the future. If we are being overly ambitious, we won’t know until we try!

Training Maddie

With this foundation, the next step was to build a training dataset designed around the IB curriculum. For this, we implemented an approach similar to the one outlined in the Baize Chatbot paper.

Simply the process is broken down into three steps:

Distill the raw data (IB curriculum books) into paragraph – question pairs using a one-shot prompt. 1
Distill the paragraph – question pairs into question – answer pairs using a zero-shot prompt. 2
Train the LoRA using HuggingFace’s PEFT library on top of the merged LLaMA and Vicuna models above for 15 epochs

For the initial test, only the MYP Language Acquisition book was used.

Maddie’s Tricks

Human: What is philosophy behind MYP Language Acquisition?

LLaMA-7B:
The philosophy behind the MYP language acquisition program. (Anchor provided to keep it on track)
The aim of this course is to provide students with an understanding and appreciation for a second or third foreign language, while developing their skills in listening, speaking, reading and writing through authentic materials. Students will be encouraged to communicate effectively using target languages as well as develop critical thinking abilities that are necessary for effective communication across cultures. Throughout the year they will have opportunities to participate in class discussions about topics related to culture, society, literature, art history, science, politics etc., which will enable them to express themselves clearly and confidently both verbally and in written form.

Vicuna-7B:
Philosophy of language acquisition in the MYP program focuses on developing students’ ability to communicate effectively and efficiently. The goal is for them to be able to express themselves clearly, both orally and written. It also aims at helping them understand how languages work so that they can use it as an effective tool for communication.

Maddie-7B:
The philosophy of MYP language acquisition is to enable the student to become a critical and inter-culturally aware communicator. It encourages the development of multiliteracy skills, which include not only literacy in textual forms but also visual, spatial, audio and video. Additionally, it fosters an awareness and understanding of different cultures leading to personal development and cultural identity. Therefore, this subject provides opportunities for students to explore their own culture while reflecting on the perspectives of people from their own and other communities. Furthermore, through these explorations, they develop confidence as principled, lifelong learners by making responsible decisions and taking action.

Human: How can inquiry-based learning activities be incorporated into an MYP Language Acquisition class?

LLaMA-7B:
Inquiry based learning can be (Anchor provided to keep it on track)
1) a process of discovery, and/or 2) the development of knowledge. What are some examples that you have seen in your own teaching or other teachers’ classes where these two definitions were evident? Please provide evidence to support this claim (either from personal experience or research).
Inquiry is defined as “a questioning attitude” by [https://en.wikipedia.org/wiki/Inquiry_%28education%29 Wikipedia]. This definition implies that students must first ask questions before they begin their investigation; however, I believe it also means that there should always be a sense of curiosity within every student. Curiosity is something we all share at one point or another throughout our lives, but not everyone has been taught how to harness its power for educational purposes. For example, my daughter was recently asked what she wanted to do when she grew up. She replied with “I want to be a teacher!” When her grand mother heard this response, she immediately responded with “That’s great! You will make lots of money.”

Vicuna-7B:
Inquiry-Based Learning (IBL) is a teaching method that encourages students to ask questions, explore answers and develop their own understanding of concepts. IBL activities are often student centered with the teacher acting as facilitator rather than lecturer or guide.

Maddie-7B:
Inquiry-based learning (IBL) is a pedagogy in which the teacher facilitates student inquiry, directs them to explore real-world problems and encourages them to develop new understanding. Therefore, inquiry-based learning can be incorporated into an MYP Language Acquisition by allowing students to identify topics of personal, local or global significance and ask questions about these topics. Through this process, they can develop theories and make connections between their inquiry and the target language. Additionally, IBL often involves complex and sometimes ambiguous problems that require sustained inquiry for extended periods of time. Therefore, MYP units of work should provide long-term opportunities for meaningful exploration of linguistic concepts and contexts. The key concept “Inquiries into language acquisition and culture encourage the development of appropriate strategies for further language learning”. This means that inquiring into language acquisition and culture can help students develop the ATL skills necessary for future language learning. Thus, inquiry-based learning can enable students to develop their language skills while also exploring issues that are personally, locally or globally significant.

Human: In MYP Language Acquisition, how are students phased?

LLaMA-7B:
Students are phased (Anchor provided to keep it on track)
into the language acquisition course based on their English proficiency.
Student: I have a question about my placement in the MYP Language Acquisition class. How can you tell if someone is ready for this level of instruction or not? What do we need to know before coming here?
Teacher: We look at your previous school records and test scores when determining which levels would be best suited for each student.

Vicuna-7B:
The phases of language acquisition in the MYP include the following:
Pre-production phase – Students develop their understanding and knowledge about a new language through listening to it and observing its use by others. They begin to recognize patterns within words and sentences as they learn more vocabulary and grammar structures. This is also known as the “silent period”.
Production Phase – The student begins speaking or writing using what he/she has learned during the pre-production stage. At this point, mistakes may be made but these should not hinder progress.
Interpretation Phase– Here, the learner becomes familiar with different cultural aspects related to the target language such as customs, traditions, etc. He/She can now understand the meaning behind phrases and expressions used in context.
Comprehension Phase – The student is able to comprehend the full range of meanings associated with the target language. Now that he/she knows the rules for pronunciation and structure, he/she can communicate effectively both verbally and nonverbally.

Maddie-7B:
Students in MYP language acquisition are grouped into phases based on their experience and ability in the target language. The phases are:
Phase 1 for beginners with no prior knowledge or limited knowledge and understanding of the target language;
Phase 2 for beginners who have a basic knowledge and understanding of the target language but require working knowledge and application of some grammar and vocabulary;
Phase 3/4 for intermediates who have an understanding of most grammatical concepts and can use the target language to communicate and interact in a range of familiar and unfamiliar situations; and
Phase 5 for proficient communicators who can transfer their linguistic skills and apply them in different cultural contexts and for a variety of purposes.

While Maddie is better able to articulate general ideas about the IB, it still struggles with specific details. Furthermore, with the tendency to both hallucinate and generalize, it is not yet reliable enough for tasks in school. But it is still early days!

Next Steps

The next steps we are taking to help grow Maddie are:

(Training) Curating a more comprehensive dataset derived from multiple IB curriculum docs as well as open-source/CC resources to better align the general tone
(Contextualizing) Incorporating a separate process of vector embeddings so that Maddie can reference both IB documentation and school curriculum docs
(Reinforcement Learning) Developing an RLHF pipeline using HuggingFace’s TRL library

Paragraph/Question Distilliation One-Shot

{
    model: "gpt-3.5-turbo",
    max_tokens: 2000,
    temperature: 0.8,
    messages: [
      {
        role: "system",
        content: `The user will provide you with a paragraph from the IB Curriculum Guides. 
          Your task it to write five questions of growing complexity that can be answered with information from the paragraph provided. 
          The questions should go from very simple summarization, to identifying specific details, to infering meaning.
          Do not preface questions with phrases like "In the paragraph it says..." or "According to the paragraph...".
          Return your response as an array of five strings.`,
      },
      {
        role: "user",
        content: `MYP Language Acquisition Guide:   The ability  to communicate  in a variety  of modes  in more  than  one  language  is essential  to the concept  of  an international  education  that  promotes  multilingualism  and  intercultural  understanding,  both  of which  are central to the IB's mission. All IB programmes  require  the students  to study,  or study  in, more  than  one  language  because  we believe  that  communicating  in more  than  one  language  provides  excellent  opportunities  to  develop intercultural understanding and respect. What is an IB education? (2013, updated 2017) The study  of additional  languages  in the MYP  provides  students  with  the opportunity  to develop  insights  into  the features,  processes  and  craft  of language  and  the concept  of culture,  and  to realize  that  there  are  diverse ways of living, behaving and viewing the world. The acquisition  of the language  of a community  and  the possibilities  to reflect  upon  and  explore  cultural  perspectives of our own and other communities: \u2022 are valued as central to developing critical thinking, and are considered essential for the cultivation of  intercultural awareness and the development of internationally-minded and responsible members of  local, national and global communities \u2022 are integral to exploring and sustaining personal development and cultural identity, and provide an  intellectual framework to support conceptual development \u2022 greatly contribute to the holistic development of students and to the strengthening of lifelong  learning skills \u2022 equip students with the necessary multiliteracy skills and attitudes, enabling them to communicate.`,
      },
      {
        role: "assistant",
        content: `['Why is it important for IB programmes to require students to study or study in more than one language?', 
          'What are some benefits of studying additional languages in the MYP?',
          'How do the acquisition of a language and reflection upon cultural perspectives contribute to developing critical thinking?',
          'In what ways do language acquisition and exploration of cultural identity support conceptual development?',
          'How do multiliteracy skills and attitudes obtained through studying additional languages prepare students for communication in a global community?']`,
      },
    ],

    {
        role: "user",
        content: <new content>
    }
  };

Question-Answer Distillation Zero-Shot

{
    model: "gpt-3.5-turbo",
    max_tokens: 2000,
    temperature: 0.8,
    messages: [
      {
        role: "system",
        content: `The user will provide you with a paragraph from the IB Curriculum Guides and a question. 
        Answer the question using information from the paragraph.
        If you cannot answer the question, output 'I cannot answer this question.'`,
      },
    ],
  }