Measuring Human Return

Understand and Assess What Really Matters for Deeper Learning

Measuring Human Return responds to the issues facing educators today in their work to develop the outcomes that matter. In addition to introducing and describing the deeper learning outcomes (self-understanding, knowledge, competency, and connection) as those toward which to focus all efforts in our school systems, MHR provides the tools, approaches, and framework for developing these outcomes alongside the professional capabilities required to bring them to life.

Check out the book

The Audio Companion Guide

By Mary Coverdale, Executive Director of The Learner First, Australia

The Audio Companion Guide is a free resource intended to support educators and leaders to dive into Measuring Human Return. It offers insights and practical applications.

Below you will find each episode as well as show notes.

Chapter 1

Outlines the concepts and beliefs underpinning The Learner First work.

Chapter 1 – From Surface to Deep

Question:

Do you believe that you are able to enable every student to be successful by shifting the way you assess and measure their knowledge, skills, and understanding?

“The global aim of humanity is to live in ways that are meaningful and fulfilling” (MHR, pg. 3).

The Learner First has created a dynamic road map to gather evidence to use to work towards clarity around the causes of underachievement. Deeply understanding what works, for whom, and why is the core focus for professional learning opportunities for teachers. “Measure” and “develop” are factors in a synergistic relationship—they enable teachers to work with learners, parents, and communities to rethink the needs of students in a complex, changing world. The core deeper learning outcomes of Self-Understanding, Knowledge, Competency and Connection drive the development of skills to contribute purposefully to the lives of others in a meaningful and fulfilling way.

In a world which is no longer binary, the concept of what success means has shifted into an understanding that broader outcomes are valued by society at large. Traditional metrics don’t measure the whole child and so are less equitable in terms of catering for the diversity of children in our schools.

Refocusing and re-centering in order to determine, with precision, the needs of all students, following a thorough identification of assessment and capability gaps, enables them to learn, create, act, and succeed.

Collaborative change through authentic formative inquiry focusing on the work that matters is key to the moral purpose underpinning identifying, assessing, measuring, and then developing the right work at the right time for the right students. They engage with the learning and they engage with others, finding out who they are, what they know, what they can do, and what they can connect with and contribute back to others. Intrinsic motivation is activated.

Authentic intentional practice is focused on assessing and measuring the breadth of available evidence. This is then synthesized to fully inform understanding and then practice.

“Assessment provides those individual points of evidence that combine to inform the measurement process” (MHR, pg. 2).

Strength and diversity of assessments is key. Comprehensiveness enables an authentic reflection of an individual. Deep learning begins with a shift in thinking, particularly in terms of biases and assumptions—some deeply held beliefs can create barriers to this work as teacher values and attitudes need to be examined and informed in order to change expectations and teaching and learning practices. The Culture Iceberg exposes the levels of alignment in an organization which must be acknowledged and challenged in order to gain traction for this work.

Seven Assumption Shatterers begin the journey. The Change Team can explore the “pulse” of the staff cohort by unpacking these at staff meetings.

  1. Learners aren’t failing in the system, the system is failing learners
  2. The only acceptable mindset is 100% success
  3. Equal treatment leads to inequitable outcomes
  4. Focusing on the least-served learners creates shifts that benefit everyone
  5. Systems can measure what matters for their learners
  6. Students are partners in instructional practice
  7. The journey to system wide deeper learning is collaborative

A shift in mindset can support teachers to:

  • Improve the outcomes of a struggling learner be recognizing a failure to meet his needs,
  • Collaborate with learning partners around the outcomes that matter,
  • Get to know the learner as an individual and personalize the learning around his interests and needs, and
  • Build a classroom community rooted in newfound understanding of what it takes to change learners’ outcomes.

There are Five System Capabilities which form an evaluative snapshot, analyzed and synthesized in alignment with work focused on the Assumption Shatterers and the Five Frames of Measurement:

Engagement, Development, Clarity, Inquiry, and Depth.

“Inquiry describes the ubiquitous, continuous process of assessment, design, implementation, measurement, reflection and change” (MHR, pg. 6). This pervades every level of a deep learning system, framing all activities within it.

In this narrative, “assessments are both evidence and solutions.” They evidence where learners currently are with their learning and what they need to do to progress further. They are also the subsequent solutions—new evidence is collected about new learning levels and needs.

AMMA is Authentic Mixed-Method Assessment. It describes the process of gathering the full range of evidence, both qualitative and quantitative, required to arrive at a fully formed understanding (measurement) of overall levels of progress and development. Single-narrative thinking has no place in this way of working.

It’s about gathering evidence of what’s important with purpose and intention and so developing “the full range of outcomes that humanity values and that all learners need for success” (MHR, pg. 7). The set of physical and cultural conditions can be described as:

  1. Partnerships with students, parents and others
  2. Natural and built learning environments
  3. Digital tools and other technologies
  4. Individual inquiry comprises elements of authentic practice.

Equitable, personally relevant learning experiences, every day, no matter the content or aim of the learning at hand effectively engage the student and the teacher.

Key questions for the Change Team, Leaders, and Teachers are:

Do our/my objections, conflicting interests or system constraints prevent our/my students from deeper learning? Do they advance or inhibit what’s best for learners? Are we/am I aligned with the needs of my learners? How do we/I know?

Can we articulate why we are embarking (or will embark) on the deeper learning journey? How does this journey align with our vision, values and our AIP? How will we develop “buy in” across our entire community?

Chapter 2

Outlines the 5 key capabilities used to develop an evaluative snapshot of your school or system to enable an evidenced improvement plan to be developed

Chapter 2 – System Capabilities

Question:

Do you have a powerful understanding of what constitutes underachievement for every student? What learning do you have to learn to become proficient in this capability?

“Healthy systems where deeper learning is fostered, developed, measured and sustained for all, continuously develop the capabilities that best support the development of every student.”

There are 5 key capabilities:

  1. Understanding Your System
    1. Establishing clarity around the real causes of underachievement
    2. Developing a systemic professional learning strategy that addresses the real causes of underachievement
    3. Understanding what works best, for whom, and why
    4. Gathering meaningful, collective evidence in the system and in the community

The Capability Discussion Starters catalyze exploring and synthesizing these dimensions. Honesty in acknowledging where the school sits across the dimensions is vital to determining what evidence is required to get clarity around the true causes of underachievement. This  helps to authentically identify the learning opportunities required to begin the TLF journey.  The journey of Burlington Edison School District (BESD) supports developing an understanding of what the journey can look like.

They began an 18 month journey which began with commitments to:

  • Creating a culture of learning
  • Developing key learning competencies, such as communication, collaboration, creativity, and critical thinking
  • Ensuring equity through high-quality practice
  • Valuing the cultures and identities of each student and family; and
  • Engaging families and the wider community in students’ learning and success

Their staff also created 10 Learner Attributes which when built were enlarged to identify demonstrated evidence.

“No matter the level of the system at which you’re working, the key is to establish alignment between what your system demands and what your learners do” (MHR, pg. 29).

Measurement is at the root of that alignment. There is a rubric for each capability (Appendix A). Following this “audit,” a plan will be developed for each capability—this comes from knowing where you are and what you need. Measurement shows you the capability gaps. If the evidence is lacking, measurement will show you the assessment gaps that need to be filled. Both are vital.

Key questions for the Change Team, Leaders, and Teachers are:

After undertaking measurement of the four dimensions of the Understanding Your System Capability:

  • Record your evidence used for the decisions.
  • Determine an overall rating for the Capability using the five-point scale.
  • Synthesize the information using the guidelines.
  1. Engaging Learners, Parents, and Communities as Real Partners
    1. Partnering in every aspect of the inquiry process
    2. Working with parents to jointly discover, understand, and realize learners’ needs and goals
    3. Supporting, valuing, and utilizing partner insights and engagement
    4. Providing opportunities for technology-enhanced, connected learning anytime and anywhere

Drilling down into what REAL engagement and partnership require often reflects that the depth and breadth of practice aren’t always where they could be. Real partnerships leverage the knowledge and abilities of all parties in an equitable way that leads to learning for everyone. For deep learning, students should partner with their peers, teachers, families, and communities both local and global. Technology is an enabler for broad relationship development that supports the goal of being “citizens of the world, committed to positive relationships that push thinking and create real change” (MHR, pg. 33).

Key questions for the Change Team, Leaders, and Teachers are:

  • On each level of the rubric, determine where your system/school sits (four dimensions).
  • Record the evidence, then synthesize to determine an overall rating for the capability.
  • Identify the evidence used to determine the ratings. Do you need to identify more evidence for any aspect of the rubric?
  • What do your ratings tell you about where you are at and where you can progress?
  1. Identifying and Measuring What’s Important

Question:

What, in your view, is important to be assessed and measured when determining the progress of students? How do you involve learning partners in assessment/ measurement? Are you committed to seeing past the standards to fully see the learner? Do you deeply understand what your learners need to be successful? What does success mean?

“Measurement—it’s a driver of individual and system change centered on everything that makes our learners who they are. It’s all about the learner all the time. Measurement relies on knowing and capturing what’s important for students” (MHR, pg. 38).

Evidence comes from assessment. This allows learning to be measured in a valid and reliable way. Authentic assessment practice involves establishing an evidence-based culture necessary for accurately and fully informing the measurement of student learning.

Identifying and understanding what is valued, you then develop and build clarity around deeper learning measures and then use them within an ongoing inquiry process towards depth.

A comprehensive measurement framework begins with an evaluative snapshot and synthesis of the current work:

  1. Collaboratively identifying what success looks like for your learners.
  2. Establishing clarity using deeper learning tools
  3. Engaging in authentic assessment that fully informs the measurement of learning
  4. Moderating exemplars to ensure inter-rater reliability and to identify and spread best practice

BESD asked, “What do you want to know and assess about your students”? They used community input to determine and define Four Core Covenants. The tools which have emanated from this “define” the school. (Character, Commitment. Leadership, Integrity.)

Identifying the individual components of what you want to measure is vital—precision around the points of learning enables precision in assessment and measurement. Consider each deeper learning outcome, then use the Learning Development Rubric to measure where you are in relation to the development of each outcome.

Key questions for the Change Team, Leaders, and Teachers are:

Reflect on and write down the specific outcomes you want for your learners, grouping them under each respective deeper learning outcome. This exercise should be done in partnership. Define self-understanding, knowledge, competency and connection. What makes up each of these outcomes and what does it look like when learners have developed them?

Juxtapose these lists with the Learning Development Rubric.

What are your school’s/system’s current levels of progress in relation to each of the DL outcomes?

What evidence are you using to determine your ratings? How and in what ways can your system evolve?

“Deeper Learning Outcomes”

The development of the five capabilities enables learners’ development of the four Deeper Learning Outcomes: self-understanding, knowledge, competency, and connection. They’re the components of a meaningful and fulfilling life no matter where we are in the world, because they’re the outcomes that help us contribute back to others’ lives and the world. They’re what we need to help our learners develop in schools—not just in mission or theory, but in purposeful and intentional practice. Our reasoning for the development of these outcomes can be summed up as follows:

  • Humanity demands that everyone has the opportunity to lead a successful life marked by meaning and fulfillment.
  • Therefore, education has to support learners in this process of personal, collective and global development—it’s our moral, progressive and evolutionary imperative.
  • In order to fully support this growth, everything we do as educators and learning partners working within and in partnership with education systems has to be tuned to developing the outcomes consistent with how we as humans find and sustain success.

Using the Learning Development Rubric, ascertain where your Change Team, Leadership Team, teachers, students, and parents believe your work sits on the five-point scale. Establish a whole-school position. Record this.

  1. Leading for Deep and Sustainable Change

Question.

What leadership attributes do you possess which, when combined with those colleagues and students you work with, enable the development of deeper learning outcomes?

Effective leaders know how to live and communicate deep cultural change which supports all members of their community. They are able to articulate what is needed to thrive and they work collectively to embed and sustain deep cultural change. They are learner focused, challenging practices which are not aligned to positive growth for all. They foster lifelong leadership at all levels. Our leaders need to develop the full range of learning outcomes, to be “leaders who not only know what’s important but also are committed to intentionally developing and therefore measuring it.”

Dimensions of Leading for Deep and Sustainable Change

  1. Focusing every action and decision through the lens of the least-served learners
  2. Prioritizing what needs to change and collaboratively designing solutions
  3. Leading change all the way through to measured, sustainable outcomes
  4. Fostering and supporting student, teacher, system, parent, and community leaders

Through effective leadership, “deeper learning is an embedded and authentic mindset that enables and enhances daily practice.” Deeper learning measures and other tools are leveraged to develop shared language and understanding of DL. Play is central to the concept of creative thinking being enabled to develop new ideas and practices, run with them, learn and build from their outcomes, and celebrate successes. Student leaders are enabled to take action to seek solutions to local and global problems.

Leaders accept responsibility when they aren’t doing enough to make deeper learning outcomes a reality.

Key questions for the Change Team, Leaders, and Teachers are:

  • Use Appendix A4 to determine where you are currently on this capability.
  • Record the evidence you used to make that decision.
  • Synthesize this to determine an overall rating.
  • What evidence are you using to determine this rating?
  • Do you need more evidence? What do your ratings tell you about where you are and how you can progress?
  1. Creating a Culture of Learning, Belonging, and High Expectations for All.

Question:

At my school, in my classroom, do all students feel valued and respected? Can I identify, and then support, my least-served learners? How?

“It’s our responsibility to develop cultures that instill a sense of belonging in all learners so they never have to—and never want to—hide who they are, what engages them, troubles them, and how they can contribute within their classrooms, their schools, and beyond.”

A positive culture of learning is characterized by innovation, reflection, and a willingness to try new strategies, make mistakes, reflect, and learn to ensure deeper learning flourishes. Belonging and High Expectations are core to this capability. Belonging requires genuine equitable relationships in which learners truly feel they’re expected to, and believe they can, succeed on the strength of their capabilities and connections. Expectations that are deeply felt are high expectations because they are centered in relationships.

Dimensions of Creating a Culture of Learning, Belonging, and High Expectations for all are:

  1. Providing the freedom to learn, share, celebrate and improve
  2. Engaging in deeper learning experiences and teaching for 100 percent success
  3. Embedding a deep commitment to and expectation of success for all learners
  4. Cultivating genuine personal relationships
  5. Celebrating students’ learning and identities everywhere and in all things

Connecting our school/system to the notion of “humanity” is vital to shifting culture—building personal relationships, honoring social and cultural diversity through differentiation, knowing the names and faces of the “underserved” students by including “all students” in strategies for improvement, identifying the special talents of all students, and understanding that the focus of the work extends beyond the school fence into the wider community.

Key questions for the Change Team, Leaders, and Teachers are:

  • Use Appendix A5 to determine where you are currently on this capability.
  • Record the evidence you used to make that decision.
  • Synthesize this to determine an overall rating.
  • What evidence are you using to determine this rating?

Do you need more evidence? What do your ratings tell you about where you are and how you can progress.

Chapter 3

Outlines the Role of the Change Team.

Chapter 3 – The Change Team Process

  • Collective Cognition for Collaborative Change

Deep change is collaborative, and to embrace it in practice means that the depth and breadth represents, for many, a change in “the way we do things.” Deep, sustainable change doesn’t happen when people are trying to work in pockets in schools or systems and it can’t happen as a result of individual action.

It requires collective commitment to the vision and the process for change. It takes involving all partners in learning—at all levels—and, in particular, our students.

The vehicles are Change Teams—connected, multilevel teams that ensure that the ideas and practices that have a positive impact on learners are fostered, spread, supported, and developed. Change teams track and measure progress, identify areas of success and areas for improvement, and drive necessary changes from within as well as to any level of the school system. The Capability Rubrics (which I have outlined and you can access as Appendix A from the Corwin website) enable deeper learning and so play a major role in the work of the change teams.

In our system we have schools, regions, and Education Departments. Change teams can exist within and between every one of these levels. The connected and collaborative change team aligns the system vertically and horizontally—from learners to leaders, and between schools and school systems. They build alignment and consistency in ways of thinking, knowing, and doing. Any school aiming to embed deep learning through the frameworks can form a successful change team with the local resources and learning partners at their disposal.

Members of a change team can include:

Students – They are the central focus of the change, so they can play a central role in driving the change. Their engagement will be determined contextually. Student focus teams may be formed to advise a whole-school team. Any opportunity to have a meaningful voice, share opinions and insights, and gather feedback will support the actions of the change team—in awareness, influence, authenticity and integrity.

Teachers – This will include teachers of the change team students as well as those with a capacity for or a history of leading change effectively.

School Leaders – The principal will always be on the change team and is likely to play a lead role in selecting change team members. There may be other school leaders on the team as well. They  may be coaches, administrators, Leading Teachers, etc.

Parents and Family Members – These people are the leading experts on their children and know what matters to them. At least some of these should be related to the most struggling learners, since their insight is vital in terms of why the school or system hasn’t worked for their children and how it can improve to do so.

Community Members – Representatives who can voice the needs, concerns and values of the broader community and who can offer expertise on what students need in order to be successful. They form an essential link between the school and the community.

A Regional Representative – These representatives form the bridge between the best practices identified at one school and the other schools in the region. As well, they can remove barriers to the changes that are identified and help them to be implemented.

Each level of the system can form change teams, including in the top ministries. The principles of representative inclusion in the teams remains the same. Hearing the voices of those involved, making sense of circumstances in each context, and honoring the role that can be played in change management.

The line of sight between each level of the system, all the way to learners, can collectively refocus the work you do through the lens of the learners who are struggling the most. Remember that one of the Assumption Shatterers was that focusing on the least-served learners benefits everyone. Rethinking assessment practice through the eyes of the least served students and collectively determining what they really need in order to be successful, frames the formation and operation of change teams at every level.

At the classroom level teachers gather evidence about students and their level of learning to identify those students who are struggling the most. The tools support a holistic approach to getting to know the students and their parents deeply. The learning can then be designed to meet their needs, sharing that with, and using the collective intelligence of, the change team. Based on learning from direct work with the students and from the students’ own input, the collaborative output is welcoming, nurturing, and deep. Each school change team focuses on building clarity about what needs to change and why, removing barriers and creating conditions that enable deeper learning that is inclusive of all learning partners.

This translates directly to the way that the region and school systems can do their work to build capacity at their level and to align with the changes occurring at the grassroots level, supporting the schools by enabling policies, resources, service provision, and measurement and assessment systems.

All decisions at all levels are considered in light of their impact on the full range of deeper learning outcomes.

An ongoing  inquiry approach is used in each change team to know, understand and respond to the needs of each learner. Figure 3.2 on page 63 reflects this fluid and continuous process of learning assessment, evidence gathering, design and implementation, measurement, reflection and change which shapes the deeper learning work.

The change team can use inquiry to develop the five system capabilities. The questions are asked and answered in each rubric to assess the extent that you know your school or system and its learners, the extent to which you engage all stakeholders as active partners, how you identify and measure what’s important, how you lead change, and to what extent you’ve developed a culture where all learners can succeed.

Synthesizing that evidence, you then measure and reflect on current levels of development, then change practice and process to address identified needs. Solutions are collaboratively designed and implemented providing opportunities to assess and measure progress, identify new learning and needs, and design and implement additional solutions to respond to the real-time needs of the change team members and their learners.

Evidence for your school capabilities needs to be the right evidence—precision and truth are vital. Do you really have enough information to make a fully evidence-based judgement? You do get better the more you do it—you build your understanding of the measures to identify and demonstrate the importance of a given outcome or capability that you haven’t looked for or previously designed. You find evidence that is intentional, meaningful, and deliberate. This is the vehicle for your change plan. You work out the relationship between assessment and evidence, between assessment and measurement, and between all elements of the inquiry process. You work out how Authentic Mixed-Method Assessment gives you a deeper understanding of your students your systems and your assessments.

When your change team collaborates on this activity, it creates the collective cognition or intelligence that enables a way forward. Mindsets shift, what’s needed to improve student outcomes is established—the collective gives us a micro and macro view of the work.

The Change team, using inquiry, levers the mobilizing and sustaining of change, develops a collective purpose, and, most importantly, focuses all actions and decisions on learners’ outcomes.

Chapter 4

Outlines the frames of measurement, the deeper learning outcomes and the elements of authentic practice

Chapter 4 – Engaging Partners around What Matters

Beginning of Section 2 of the book—focus on building engagement, developing capacity, embedding clarity, and deepening outcomes.

What assessment enables measurement of what matters? How do we work out what we assess?

Identifying and measuring what’s important, being systematic about measurement, is a key lever for developing each of the capabilities that lead to depth in practice—and so depth in student outcomes. Empowering all learning partners, new and future, is catalyzed by using Five Frames of Measurement that set up and sustain your work:

Engagement – of all learning partners to identify and describe what’s actually important for learners and what contributes to their success.

Development – using the descriptions from the engagement phase, filling measurement gaps, developing rubrics, learning progressions, guides, or other measures and tools that collectively measure and develop deeper learning enablers and outcomes.

Clarity – of shared language and understanding of deeper learning among all learning partners, as well as aligning the framework with your school and system.

Inquiry – as a process is embedded in your daily, strategic, and overall practice, designing, implementing, and moderating assessments and making authentic, evidence based-decisions through the lens of learner impact.

Depth ­– occurs when you leverage your new learning to continuously deepen your measurement, ensuring your students remain at the center.

These frames form an exciting fluid journey—they aren’t linear, however they give you an opportunity to operate simultaneously through each lens so that as new partners emerge they can be included; as new measures and tools are developed, you deepen the work. You continuously build capacity and you continually monitor, respond, measure and improve as part of the inquiry process.

The first part of the student-focused work is to develop an evaluative snapshot—knowing your individual students, who they are, what’s important for them to know and be able to do, and in what ways it’s important for them to connect with others in the world. You’ll know a great deal about your students, particularly if you’ve already completed the depth in practice exercises in Chapter 2.

Building from this to get it right will require engagement with all learning partners to garner perspectives and understandings that will constructively widen and deepen your knowledge. This initial engagement forms your evaluative snapshot—a complete overall picture of your learner, the context, what’s important, identified strengths and needs—these lay the foundation for change plans.

These snapshots may involve:

  • Interviews, surveys, and/or events that involve students, parents, teachers, community members, and any other appropriate learning partners.
  • Formal reviews of policies, environments, systems, assessments, measurements, standards, technology, curriculum, parent and community engagement, unit or lesson design, as well as leaning experiences.
  • Analysis of achievement, well-being, behavior, attendance and other data that will support the development of a deep understanding of the student.

This data will support your understanding of the types of assessment evidence that can and will inform the measurement of your capabilities. These will drive your ongoing evaluation of where you are in relation to where you want to be.

If we use the inquiry process, the evidence enables us to design and implement  appropriate assessments that give the precise information we need to measure capabilities then reflect and change.

Measurement is a change lever, so if we can identify exactly what we consider to be important to measure we will effect shifts in assessment.

So, engagement is the first phase of the whole-school inquiry process—it’s where you develop clarity about where you are and where you need to be. Figure 4.1 on page 73 of the text gives you a simple snapshot of what it looks like figuratively to synthesize assessment evidence to make capability ratings to determine strategies to improve. You create assessments that help you measure what matters.

Your change team is critical to the work. A decision-making body with a collective deeper learning focus, these teams are a capacity building body and are created to suit the context of the school.

The Burlington Edison model gives you a powerful insight into the understanding of Drucker’s quote about culture eating strategy for breakfast. Utilizing the deeper learning tools to stay focused and using a values-based approach to risk taking, the district was able to drive the change including deep dives into what success really looks like and being empowered to authentically shift understanding and practice. The Change teams move the school/system from the current picture to the desired picture of success.

Contextually and culturally evaluating the school or system provides actionable evidence of what the learning partners value. This is then used as a platform for application in measurement tools which then capture the complete picture of succes—not just the academic outcomes.

Using inquiry, the change teams work to understand their schools, their learners, and what’s important. They engage learning partners around deeper learning, leverage tools to measure progress and success, create a culture that fosters deeper learning outcomes, and support others to do the same. It’s deep and intentional, and the eight tools in Appendix 3 in the support guide to this book give you a suite of support to kick start this work. They are comprehensive and highly supportive.

If we focus on what it means to identify and describe what matters, we come to the work-related aspect of the deeper learning outcomes, the system capabilities. and the elements of authentic practice—the tools relate to both outcomes and enablers, and are embedded in the Five Frames of Measurement to identify, describe, and subsequently measure what matters for our learners.

The reason the deeper learning outcomes have been developed is enshrined in the reasoning that:

Humanity demands that everyone has the opportunity to lead a successful life marked by meaning and fulfilment. The inference is that education has an obligation to support learners in this process of personal, collective, and global development. It’s a moral, progressive, and evolutionary imperative.

The first deeper learning outcome is self-understanding, which is about who we are, who we will be, and why. For in-school learning, this is a shift where students will have an intentional focus on understanding themselves, exploring what they’re interested in and capable of, how they can make a difference in the world, and whether they know what gives them meaning and fulfilment. Developing those understandings is central to this learning outcome, and learning at school can enable and accelerate this work. We begin this work by finding out who they really are through deep engagement.

Knowledge acquisition is aimed at simultaneous development and so is acquired and then enhanced, leveraged, and created as a direct result of students partnering in designing, implementing, assessing, measuring, and reflecting on the work. We have to incorporate the content in a way that connects with students’ lives and interests. Then existing and new knowledge will be embedded along with the other learning outcomes.

Competency looks at what you do with your learning. These support the meaningful application of knowledge inside and outside the classroom. Learning sticks when the focus is shifted from the learning acquired to its application. The competencies of character, critical and creative thinking, citizenship, communication, and collaboration have been identified as skills for success in our changed world. Measuring these is a vital step forward.

We think about our lives in relation to the connections we share with others and the world. The strength and nature  of these determines how we feel about the world and our place within it. Our lives are more meaningful and fulfilling when we share them with others and we contribute to the general wholeness and well-being of humanity. Education can facilitate and strengthen learners’ connections with their peers, other learning partners, their learning environments, and their communities across the globe. Developing citizens capable of connecting for collective change is a worthwhile outcome.

The culture of each school and system can be that every action or decision is made in light of these deeper learning outcomes. They are interconnected, they contribute to students’ capacity to contribute back, and they lead to the unique, meaningful and fulfilling lives we want for our students.

If we break down deeper learning, we have the theory of action that learners develop the necessary knowledge better while developing the deeper learning outcomes. In the past we haven’t explicitly identified these outcomes and we haven’t known how to measure them. That’s the big question—how do we measure them?

On page 81 in figure 4.5 there is a comprehensive list of those elements, developed by learning partners at Burlington Edison, that they considered important to take into consideration when considering elements of self-understanding. After identifying this profile, they then described learners who are developing self-understanding as individuals who understand:

What’s important to them; their knowledge, competencies and connections; how to reflect who they are in what and how they learn; how to deal with challenges and frustrations; what personal success looks like; how their experiences, history, and heritage shape their perspectives, their values, and the ways they view themselves and others; the purpose of learning; and how they contribute back in a range of contexts and situations.

Students thus are confident and proud, they feel like they belong, they celebrate themselves, and they understand their learning potential in terms of achieving their hopes and goals. When grouped or chunked, these attributes lead naturally to the Learner First dimensions of Self-Understanding:

Identity – understanding who we are and how we learn as individuals.

Place – understanding how we impact and fit into others’ lives and the world.

Capacity – understanding our potential  for learning, progress and success.

Purpose – understanding why we learn and how we can make a difference.

The dimensions of connection are:

Interpersonal – connecting with the people we know and interact with.

Environmental – connecting with the natural and built environments.

Conceptual – connecting with our learning.

Universal – connecting with all of humanity and the world.

We think, talk about, strive for, and experience self-understanding and connection on a regular basis. We bring our own insights into what they feel like. However, our experiences limit our understanding. It’s through our partnerships with our peers, teachers, parents, families, and communities that we can gain a deeper, collective understanding of any idea, item, or outcome at hand. Bringing them to life means exploring them through each of the deeper learning outcomes. The putting Depth into Practice exercise in this chapter help deepen your understanding of each dimension by bringing them to life through your own experience.

I’ve outlined the competencies which are regarded as those identified globally for student progress. They include:

Character, with a focus on the traits of perseverance, resilience tenacity and grit.

Citizenship, with a focus on global citizenship, focusing on global issues, appreciating diverse values and world views, and demonstrating an ability and interest in solving ambiguous and complex problems that impact human and environmental sustainability.

Collaboration means working interdependently and synergistically in teams with strong team-related skills, including effective management of team dynamics and challenges, making substantive decisions together, and learning from and contributing to others’ learning.

Communicating effectively using a variety of styles, modes, and tools tailored for a range of audiences.

Creativity means taking initiative and having an entrepreneurial eye for opportunities, asking the right questions to generate novel ideas, and developing the leadership skills to pursue those ideas and turn them into action.

And critical thinking means you can evaluate information and arguments, you can see patters and connections, and you can construct meaningful knowledge and apply it in the world.

We can look closely at collaboration as an example:

The dimensions listed on page 85 represent the qualities we want for our learners. The learning partners identify the language with which to describe these competencies, and it becomes central to the work. These competencies are connected and interdependent—they support, rely upon, and build one another into a coherent outcome of meaning and fulfilment marked by positive contribution. The exercise on page 86 takes you through how to link each of the competencies together for sense making in the way that each develops and enables the others.

We use what we call elements of authentic practice, four of them, to make it all happen. They are:

Partnerships – engaging students, teachers, parents, community members, and other learning partners to enable and deepen learning.

Environments – leveraging where, why and how we learn in a range of natural and built environments.

Technology  – leveraging digital tools and other technologies to connect learners, expand learning environments, and otherwise enable deeper learning/

Inquiry – leveraging the continuous process of assessment, design, implementation, measurement, and reflection and change in partnership with learners to enable and deepen learning.

These four ways of working enable and deepen your work, beginning with authentically engaging with the learning partners in understanding what really matters, so that we genuinely measure what we do decide actually matters.

Chapter 5

Outlines the tools used for designing assessment and measurement.

Chapter 5 – Developing Measures of Deeper Learning

Authentic Tools and Measurements

We are measuring for learning for life—it’s shifting the focus of measurement from a static representation of current learning to a dynamic understanding of present success and it’s sustainability.

Presented differently, traditional measurement asks of learners, “What do we know now?” Authentic measurement asks:

  1. Who are you and who will you be?
  2. What do you know and what will you know?
  3. What can you do and what will you do?
  4. How are you connected and how will you connect?
  5. How do you contribute to the lives of others, and how will you contribute back?
  6. Have you found meaning and fulfillment?

Ultimately, it’s about leading a meaningful and fulfilling life where a lifetime of learning, progress, and success is experienced.

Think about where you are now and where you need to be—self-reflection begins the journey.

If you’ve engaged fully as outlined in the first four chapters, you will have worked out definitions and descriptions for success and know what it actually looks like in your space.

The language of deeper learning reflects progressions and continual formation—where learners are now, where they are going, and how their outcomes might be continually deepened. It reflects what it means to be human and the potential we all have for continual growth and progress. It’s the “language of depth” which sets deeper learning apart from traditional approaches.

Systems of measurement have to include measures and other tools that:

  • Embody a central focus on learners and their outcomes,
  • Describe each important outcome, capability or practice using the language of depth,
  • Shift the focus to the present and the future,
  • Develop a common language and understanding around deeper learning—its mindset, practice, indicators, and outcomes,
  • Build, in and of themselves, learning partners’ capacity to assess, design, implement, measure, reflect on, and deepen learning, and
  • Enable opportunities that inform their use.

Through the engagement phase you’ve identified the outcomes, capabilities, and elements of teaching practice that are important to you, your learners, and your learning partners. You’ve also described them. The next step is to measure them. Assessment evidence means that you are able to determine which progression is targeted and what assessment evidence would look like for that progression to have been achieved. The Depth in Practice exercise on page 93 asks you to consider the Appendix D learning progressions across Self-Understanding, Connection, and Collaboration. The task asks you to dive into first appreciating and understanding how this work fits with your current ways of working and then to look at what might constitute assessment evidence for some of them, in much the same way that this is outlined for the Place dimension of Self-Understanding on page 92.

This is the essence of the shift that teachers will make in their practice. These learning dimensions demonstrate that what we know and understand about our learners and their needs comes together to support both the measurement and the development of deeper learning outcomes. We know our students and we know how to help them grow by creating assessments that lead to evidence which can authenticate the way we arrive at measurement.

Working in the dimensions will reflect that the language used will shift through the increased lens of concepts and ideas such as awareness, perception, viewpoints, perspectives, culture, family, history, belonging, and respect, and that they exist explicitly and implicitly all through the descriptions. They are products of engagement and the partnerships it fosters, and they are the products of the work.

When we engage learning partners around what matters now and throughout their lives they describe outcomes that are important to them, what they look like when they are developed, and how that development occurs. When these outcomes are identified, the descriptions are then dimensions which describe varying levels of progress. Ultimately, they are called learning progressions, which measure deeper learning outcomes. These learning progressions are built for you in figure 5.1 – Self-Understanding. This measures students’ understanding of who they are, what they’re capable of, how they impact and fit into others’ lives and the world, and how they can make a difference.

5.2 Connection measures student’s connections with others, their environments, their learning, and the world.

And 5.3 Collaboration measures students’ capacity to work interdependently and synergistically in teams with strong interpersonal and team-related skills.

The Depth in Practice tool on page 93 gives your team an opportunity to explore these tools through the lens of your own system’s capability tools, and then what it looks like to generate assessment evidence.

The way that we evidence progress through these desired outcomes and enablers is through the inquiry process. If we begin with assessment, these have to be designed and implemented to inform the measurement of learning, reflecting on and changing practice as necessary. Assessment is the object of the learning design and implementation and it’s leveraged to measure the deeper learning outcomes.

A tool that supports you through this process is the Student Inquiry Guide: Authentic Assessment (figure 5.5 Appendix E). It reflects every element of authentic practice including the fluidity and the interdependence of the elements.

It asks questions that:

  • Enable the design of the assessments that reflect who learners are and fully respond to evidenced needs and evidence gaps.
  • Embed the practices required for authentic assessment implementation.
  • Facilitate reflection on and measurement and change of outcomes and practices, in real time, using assessment evidence.

This framework is an inquiry process—you identify what’s needed and track progress and performance in real time.

If we look at a system for professional learning linked to the capability rubrics, the Professional Inquiry Guide (Authentic Assessment, Appendix F) shifts the lenses of student outcomes to the professional capabilities—we assess our own and our system or school’s learning in order to better develop our students’.

It’s a similar story with measurement.

The Authentic measurement guides will support you to use all of the evidence gathered in the design and implementation of assessments in order to measure your own or your students’ levels of learning. Figure 5.6 Appendix E2—the student inquiry guide, authentic measurement. And for the capabilities, Appendix F2—the professional inquiry guide, authentic measurement.

These will support you to match assessment evidence to the language of the measure, and provide an evidence-based understanding of where you or your learners are and where there’s room for improvement.

The design guides will help you design assessments that are purposefully designed to develop intended outcomes while also providing the learning evidence we are looking for. They will focus assessment design through the lens of authentic practice—technology, partnerships, environments, and inquiry. Implementation, reflection and change are deeply embedded in each of them.

The student guides are for use alongside the learning progressions and the professional guides are used alongside the capability rubrics.

Measuring the outcomes for these is supported by the Learning Experience Rubric (Figure 5.7, Appendix G). This supports the collaborative moderation activity (Chapter 10) which engages the learning partners in the discussion, sharing, and subsequent measurement of teaching and learning focused on the elements of authentic practice. This process enables the specificity of focus on which practices develop which deeper learning outcomes. These can then be illuminated, shared, and strengthened.

An activity to engage in more deeply understanding this work is to undertake the Depth in Practice exercise on page 113/114 to explore how two jurisdictions have undertaken and applied this work in their own contexts.

At this stage schools and systems can use the tools outlined in this text. However, some may wat to adjust to better suit the language and processes of their context. The stories of Oakland Unified School District and Oklahoma City Public Schools demonstrate how the tools can be applied contextually.

This chapter has essentially been about the tools at your disposal. Chapter 6 is about the clarity you achieve through alignment.

Chapter 6

The focus is clarity of shared language and deeper learning as an outcome.

Chapter 6 – Finding Clarity in Depth

Hi everyone, I’m Mary Coverdale, the Director of The Learner First in Australia. The CEO is Joanne McEachen, and today I’m outlining for you the core elements of Chapter 6 of Measuring Human Return. It’s called Finding Clarity in Depth. We are doing this podcast as a response to COVID-19, and we sincerely hope you are all doing well in this unusual time. We hope this podcast is helpful.

Why is a shared language so important?

When undertaking transformational change, we look to shift the culture of the way we do things in each school. For that to be effective, we need all learning partners, and that does mean all—to be on the same page—to achieve clarity of understanding, depth in practice of both teachers and students, and consistency in language at a whole-school level. We know from the Burlington-Edison model that shared trust, respect, and shared responsibility for everyone’s success are keystones for effective change.

The notion of depth can vary, the how to do depth can vary, and so too the interpretation of the tools that have been outlined in MHR.

We look to measure what really matters—the skills and dispositions we know in our hearts and minds that our students need to be fully human, to be good people, to be healthy and feel successful. If we want all students to achieve this, embracing the concepts of self-understanding, connection, and competencies is vital for them to be built in to everyday classroom practice, alongside the essential knowledge that they need to be prepared to pursue a life passion or profession.

Where does deep learning fit into this?

It’s the lens over everything—it permeates the learning. It’s not an add on, it’s an outcome. Our creativity to create a DL environment is in systemizing it into daily practice so that the language is connected to and activated in the practice.

It develops and communicates knowledge, it adds meaning to our competencies and actions, and, most importantly, as humans it enables us to connect through being able to understand ourselves and other people.

Deeper learning measure enable us to describe and develop aspects of ourselves which we find difficult to articulate. We can describe the outcomes we want for our learners and we can develop them.

The language of depth is the language of progression, of formation, and ultimately of what it means to be fully human. It captures our capacity to dynamically grow, improve, and contribute. The Learning Progressions measure, record, and facilitate learner progress.

Deep learning as a concept has been around for a very long time. We know now that to truly achieve depth in practice we need a powerful set of non-cognitive skills to balance the knowledge that our systems have prioritized. For a while we lost sight of these. It’s very important to be able to name these and then to measure these as core elements of every child’s learning. Sice each student and each school owns its own context, finding a way to deep learning will look, feel, and sound different depending on your situation—your what, why and how. It means you will articulate your own theory of action , your own starting point, your own professional learning focus, and even your own definition of deeper learning. This is a space where you respect the expectations around curriculum content. However, you respect your own teachers and students in terms of how your actions, your purpose, and your meaning integrate with the development of self-understanding, competency, and connection—the core preconditions for developing agency, developing clarity of purpose, and cocreating meaning that lead to deepening learning. We all want to do it, we just needed the how—and these tools support that capacity building for both students and teachers.

So… know your outcomes, be transparent about them will all partners, assess all of those identified outcomes, and share them with peers, parents and caregivers, and communities. These deeper learning outcomes are global human development outcomes. They live and breathe inside and outside the school walls. Talking about this work brings it to life, but it takes time to learn a new language and it takes time to bring this to life.

Is it difficult to align this work to the traditional curriculum?

We’ve tried for too long to bolt on new bits and pieces to our traditional curriculum, knowing it wasn’t engaging all students in their learning. We’ve been positively motivated, we’ve tried to do the right thing with programs for capabilities, programs for social and emotional development. And we’ve created programs for students with different needs. This work we are doing creates a way of working where the elements of knowledge, self-understanding, connection, and competencies become a holistic way of working, through the design work, planning, building, and measurement of all of these factors. We come to intentionality and leadership in this one. Effective change begins when a group of people, led well, determine that the focus on a single outcome of sections of an academic curriculum is not serving the needs of our students. It drives our moral purpose. Change happens when leaders come to know that the world has changed, that the skills required to be happy and successful can be developed in school.

It’s worth looking at the work of Burlington Edison School District (pgs. 118-126) to explore the strategies and templates for design that this district undertook to shift their planning and practice to engage every child and to serve the greater human needs of every child. The templates are transferable to all schools—the language of sectors will change somewhat, but the capacity to begin with concepts, apply them to unit planning, embed elements of what the school considers important, engage all learning partners and include community experiences in the learning, and finally to shape all of this planning into an inquiry structure or model.

You align the deeper learning into or within your existing department standards, you enhance the learning experience for everyone—adults and students.

How do you begin?

It’s important to grow from the school outwards and to recognize where outstanding practitioners live within your current environment—you will know them. Creating a guiding coalition or change team is an integral part of effective change management, and we would recommend you begin this way—with a focused group of teachers who can investigate how well your school is aligning curriculum, standards, and assessment with the deeper learning outcomes that students need. It’s gestalt at its best—an organized whole that makes better sense than the sum of its parts. Curriculum is the content, and deeper learning is the framework for embedding the depth.

A great task as an outcome of Chapter 6 after you have established your areas of misalignment is to refer to Appendix D2 and go to the environmental dimension of the Connection Progression.

Look at what evidence is required—what questions would you need to ask your learners or otherwise answer in order to develop a complete picture of their progress on this dimension?

Then, what do you already know about their learning levels, and how did you know that? How might you, your learners, or other learning partners demonstrate or further develop the additional learning specific to this dimension?

Do this once, then repeat this for each or any of the other progressions. The Change Team members could take one dimension each and share ideas and feedback with one another.

Now we move to inquiry…

Chapter 7

Authentic Inquiry Practice- begins the journey of how the work is enacted through the lens of inquiry

Chapter 7 – Professional Learning and Inquiry

Hi everyone, I’m Mary Coverdale, Director of The Learner First in Australia. This podcast is a short summary and reflection on Chapter 7 of Measuring Human Return, written by Joanne McEachen and Matt Kane. Joanne is the founder and CEO of The Learner First and she and Matt cowrote the text.

Throughout this episode I will refer to some pages from Measuring Human Return for you to dive deeper into some guides and tools that support this summary.

Keep in mind that we are exploring outcomes through evidence, and that, like the diversity in any population, we are working with unique individuals. So our assessment of deeper learning outcomes—self-understanding, connection, competencies and knowledge—and the evidence for that will vary according to the goals set by the learning partners. It matters that we hold this frame of individual identity when we seek evidence—who our students are, what they know, how they connect and how they can contribute back. The concept of meaning and fulfilment through these frames remains a constant in the work.

The five frames outlined earlier in the text provide the catalyst for continuously connecting your work. Engagement identifies and describes the outcomes that matter, develops measures and other deeper learning tools, establishes clarity, and then identifies what student and teacher learning is required to achieve the outcomes.

Once these strategies are initiated, you can connect this work to examine how effective the tools are, fill gaps that are identified, and build clarity by sharing the learning that continuously adjusts as the learners and what they need to be successful in the world shifts. Because our development is continual, it’s incumbent to continually clarify new knowledge and new understanding. Joanne says that if clarity once is clarity always, you’re not progressing in the ways you need to be.

Inquiry facilitates the processes of engagement and development, the embedding of the language and processes of clarity, and the pursuit of greater depth. It’s not circular, it’s not linear—it’s fluid. The processes may be happening in any order at any time. It’s ongoing and overlapping. You live inquiry, you do inquiry—and it’s through adult learning that this occurs.

The Inquiry mapping exercise on page 134 supports your first deep dive into the way assessment, design, implementation, measurement, and reflection can change at any time in the learning process. Mapping a year where the unique goals and learning experiences are articulated with clarity is a great beginning to enable you to design an inquiry map using available assessment evidence, measure its comprehensiveness and effectiveness in relation to the goals, and then change as necessary, reflecting in your inquiry map the shift in learner goals and intended outcomes.

This activity leads you to understand the powerful link between assessment and  evidence. Authentic Mixed Method Assessment has to be understood both in relation to student outcomes and in relation to the way teachers engage in building capabilities and practice in their professional learning.

We design and implement assessments that develop outcomes and enablers and then synthesize assessment evidence to measure them. This is the heart of this work. Assessment is one element of a wider inquiry process. It can’t be truly valid and meaningful without design and implementation, it’s static without reflection and change, and it’s limited in scope if it’s not leveraged for measurement. Assessment must tell us exactly what we want to know.

We have to design and implement assessments that capture the complete picture of progress in direct relation to what we are trying to measure, changing our practice as necessary following reflection on student progress.

In our design work, we leverage our learning partnerships, environments, technology, and the full process of inquiry to enable the development and evidencing of intended learning.

We begin by looking at how The Learner First tools support individual levels within the dimensions and then also overall ratings. We become familiar with the way these can be synthesized through the capability rubrics.

Once the student interview has been completed and the focus areas from the learning development rubrics established with all learning partners, the focus of your instruction, evidence gathering (assessment), and measurement becomes clear. So too does the learning the teacher needs to undertake to accommodate this new work. This process also increases the complexity of what it means to develop an overall rating, particularly when the evidence for each dimension can vary widely.

So, structure and consistency are vital to effective synthesis. On Page 137 of MHR you will find the CORE approach for synthesizing evidence to make an overall judgement.

C is for Centre of gravity – on balance, what does the  performance picture, the evidence, tell you?

O is for Outlying ratings – where some evidence might reflect extreme differences, you use this part of the process to check the strength of your evidence to see whether these outliers move the student far from the center.

R is for reflection and discussion – step back and look at the overall picture of strengths and gaps and determine an overall rating position. Does it make sense or add up? Through discussion with learning partners, explore the reasons behind the variations.

E stands for Explanation and Feedback – identify the most important information and evidence to share, determine how best to explain the performance picture and the reasoning behind overall decisions. Cover strengths, areas of improvement, what needs to be worked on next, and how you can work on it together.

Performance always varies. Investigating the reasons for the variation and the salient focus here is looking squarely at teacher practice. What are the learning conditions that influence the variation and how can they be addressed?

Page 138 has a Depth in Practice exercise which takes you through a sample synthesis exercise to give you an insight into how you can undertake your own. Your Change team could undertake this collaboratively within a subject area and with 1-3 students  to begin the learning journey. It’s very important that your team undertakes the discussion regarding invalid and unreliable evidence to establish the beliefs and values of your team around what can be called subjective evidence. Acknowledge the barriers that can exist to this change work. Our truths become our actions, and if your team doesn’t work through developing a sound and shared common understanding of assessment and measurement, it’s difficult to work through developing and  synthesizing the progressions effectively.

We use the measures to determine ratings. It’s important to remember—

The measures are designed to describe progress levels in relation to a target.

The descriptions are a broad picture. Different learning partners may use different evidence to inform levels of progress. The student may be looking to progress an aspect of connection. The classroom evidence may differ from the student’s actions which can be demonstrated in other ways.

You can match varying pieces of evidence to fit the same description. Early in the work evidence will be “lighter” but will grow in depth as you improve.

It’s important that the measures relate to where the student is now, not where they are expected to be at the end of the year. These ratings must be determined by professionals trained to synthesize evidence. It’s the authenticity of this process which holds the integrity of the measurement.

Deeper learning requires the synthesis of a range of qualitative and quantitative data. Building clarity around assessment and the synthesis of assessment can limit discrepancies and build inter-rater reliability. The fact that judgements will differ is OK—moderating to a collective decision is what we do through sharing perspectives and understandings to come to a collective decision.

Remember, the evidence is key.

The CORE approach, Authentic Mixed-Method Assessment and the inquiry process are all about evidence. Use your measures to build clarity, language and capacity. That way your measurement will be authentic—deeper measurement is fully informed, positive, and values the individuality of each student, their essential humanity. You have the foundations for deep and authentic assessment and you can gather and synthesize it in ways that enable deeper learning for all.

Pages 142 -143 of the text focus on the way the professional inquiry guides support a deep understanding of how the capabilities are brought to life through inquiry modes and evidence. It supports your understanding of where you can begin your journey. Again, the Burlington Edison model provides an excellent exemplar of the way that these come to life (pgs. 145-152). Having a go at Activity 7.3, putting Depth into Practice, will enable you to make sense in your own context of what the work actually looks like when you use the guides to support your identification of evidence. Synthesizing using the CORE model to get an overall rating, you then use the measurement guide to reflect on the pre-ratings to enable you to identify missing evidence or low development, to enable you to design one or more solutions to these gaps. Leveraging the elements of the guides­—partnerships, technology, environments, and inquiry—will support the solution development.

Record the evidence using the assessment guide, use the measurement guide to track progress, and then use the capability rubric again to make a post rating. The reflection will then support your own effectiveness and help you work out what steps you need to take next.

Chapter 8

Inquiring into Outcomes

Chapter 8 – Inquiring into Outcomes

Assessing and Measuring Student Outcomes

Teachers involved in this work report an increased ability to achieve more precision in assessment through use of the Progressions. They learn how to cocreate a plan to reflect what it looks like and what they have to learn to move along the Progressions.

But you need to know, collectively and culturally at your school, what success looks, sounds, and feels like. The shared and common understanding of this is a precondition and a lever for this work. Ultimately, we get to measure what we know matters.

To determine deeper learning outcomes, we look at how to make the learning outcomes live—how to activate and embed them in the daily life of the classroom.

Of course, the experiences in the classroom, based on inquiry, are the vehicle for change—for design, implementation, assessment, measurement, and reflection. The tools support you in this process.

The experiences can be large or small, short or long, and may include and involve multiple intended outcomes and assessments or, particularly initially, a single aim and assessment.

A noticeable shift is valuing that all—ALL—activity can evidence learning, and so can be deemed as authentic assessment that can demonstrate deeper learning. It’s active in real time and responsive.

Learning needs and goals can change mid-stream. Your practiced-based evidence, not evidenced-based practice, will support your understanding of whether student learning needs to be enhanced or when students want to take it in new directions. You must be prepared to design and implement assessments that meet those needs in real time. Students can prove what they know in many ways.

Leading the teaching and learning in ways that will demonstrate how this is done is what this work is all about. It’s daunting to think that every interaction with a student is an opportunity for assessment. On page 158, in Figure 8.2, is a list of activities that you will probably already use in your classroom. Do you assess these with the belief that they have integrity in terms of demonstrating understanding? They have the opportunity to asses and measure deeper learning outcomes. They are familiar, they are implementation-ready to inform a complete picture of where learners are and how they are progressing, and they are, importantly, all “assessments.”

As trained professionals, your experiences and observations are evidence. Figure 8.3 reflects the relationship between qualitative and quantitative evidence. Where there is a rich mix of assessments, your professional judgements provide valid and reliable measurements of a learner’s success. When you know your work, trust your judgement.

In many ways you will be using inquiries in your classrooms, you will be supporting students in developing their self-understanding, connection, and competencies. This work provides the deeper learning frameworks and tools so that you can assess more intentionally, purposefully, and deeply, every day—you will design deeper and more authentic assessments.

It’s important to dive into the language of each of the Progressions to find out what specifically it’s asking you to measure. The individual components, for example, of the environmental dimension of connection that we looked at in Chapter 6 include the following outcomes in relation to natural and built environments: respect, balance, safety, sustainability, mindfulness, knowledge, understanding, symbiosis, impact, interdependence, and a whole lot more. There’s incredible value through each of the measures—they require you to think about the depth of the dimension and the depth of assessment required to measure the outcomes. When we think about authentic assessment we think about authentic practice where deep learning comes to life. These practices are determined through the inquiry process—designing and implementing assessments that embed a range of proven or prospective practices, measuring their effectiveness in light of their impact on learner outcomes, reflecting on that impact, and acting on that impact.

Clearly it matters that you know what practices will most suit the needs of your learners.

Perhaps consider the following questions:

Partnerships ­­– Are students engaged as active partners in the assessment of their own and others’ learning? Are the students’ partnerships with teachers, parents, community members, and others assessed?

Environments – Are students taking responsibility for their own learning and are they pursuing questions that are meaningful and important to them? Do they demonstrate the motivation and capacity to learn anytime, anywhere?

Technology – Are students levering digital and other technologies in the direct development of deeper learning outcomes, and in the activation of learning partnerships, learning environments, and the elements of inquiry? Do they see tech as an enabler, not a driver, of inquiry?

Inquiry – To what extent and how well are students assessing their own and others’ learning? Are students taking on active or instructional teaching roles in all phases of the learning experience? Are students able to draw from a range of assessment evidence to measure their progress and performance?

The practice shift is here—it’s where the students take on roles traditionally associated with teaching. This opens up opportunities for assessing their deeper learning progress, where self- and peer-assessment become authentic elements of the overall assessment synthesis. It also brings the learning to life in that the precision of knowing where to go next on the Progressions gives agency to self-assessment.

Teaching is about using our learning to enrich our essential selves in order to contribute to the lives of others—where we get our sense of meaning. The deeper the learning, the more it resembles teaching, and so the more it resembles contribution.

The Depth in Practice exercises on page 161 are terrific for reflecting on these concepts as individuals and school cohorts. You can assess your practice through the lens of the questions I’ve outlined.

You’ll find as you activate this work in the classroom that the language of the progressions and the standards will meld and become overt as you enable the students to show what they know in different ways. Your intentionality about what, why, and how students are learning and assessed puts deeper learning into action. It’s not always major—it can be a very simple and happen as a natural part of a class where the learning is personally relevant and meaningful to the student. It’s so much more than a test or a formal assessment.

I’d encourage you to work through the Depth In Practice activities in 7.3 and 8.2 consecutively. They will support and deepen your understanding of how to use the tools to approach the planning of this work. As we say here, this is where the rubber hits the road! We also say we use the KISS model of keeping it simple. Practice using a single rubric or dimension to build your capacity for measuring depth and investigating the all-important evidence that emerges in different ways as you become more fluid and flexible about recognizing real-time evidence.

Chapter 9

This chapter outlines how deep learning comes alive in practice.

Chapter 9 – Deeper Learning Experiences

We have looked at the preconditions for implementing deeper learning experiences in the previous chapters. Schools and their learning partners center their learners around  everything from the identification of the outcomes that matter to developing and building clarity and capacity around tools that support their development. Engagement with key capabilities and practices fosters the conditions required to bring deeper learning to life. They develop, and support us to measure, these essential outcomes.

This chapter is about implementation, particularly assessment implementation. With deeper learning, entry and exit points in traditional learning cycles are replaced by ongoing engagement—learners are engaged every step of the way and the assessments they engage with respond to their needs, reflect their interests, and are marked by their own design. During and after the experience, students assess, reflect on, and measure their learning, identifying ways to change or expand upon the direction of their current and future experiences. This helps to know if students are disengaging or exiting from their learning, in which case they can be brought back in.

Outcomes are critical to learning, and the practices that guide this are vital. This chapter draws on learning experiences in communities throughout the world to identify and describe the practices that are successfully facilitating the assessment, development, and measurement of deeper learning outcomes, This will support you developing your own ways of doing this work.

You understand the purpose and value of the deeper learning experiences and your context will frame your work. Your practice will apply your work. In this text we celebrate deeper learning experiences in light of the impact of their embedded practices on the learner. Pay attention to each opportunity for assessment, both of student outcomes and of professional capabilities and practices. These are embedded in the experiences. Use the activities to reflect on and more deeply engage with each experience and to celebrate the depth of learning experienced in your own school and system.

Deeper learning is a celebration of learners and their identities, of their teachers, families, and all other learning partners, and of the outcomes that contribute to their success. It’s a celebration of learners’ contributions in classrooms, schools, households, and communities everywhere, and of lifetimes of meaning and fulfilment. It’s a celebration of humanity—and of everything that recognizing our humanity can accomplish.

The first example is from Rapaura School in Marlborough, New Zealand, where the students and learning partners realized that there was an opportunity for them to further their understanding of a Māori celebration called Matariki. They linked it to the future-focused theme of sustainability, the competency of character, and the subject Social Sciences. The learning experience was designed to deepen their understanding of themselves, their cultures, other cultures, and the festivals that celebrate them.

Learning partners continued their inquiry by identifying students’ prior knowledge and understandings. These were analyzed to identify gaps and areas of need. A whole-school-day-long exploration of cultures and festivals was held where students were immersed in learning experiences designed to introduce them to the reasons behind  and diversity of world festivals. Evidence gathered from these initial assessments as well as from self-assessments was used to measure students’ current levels of learning and inform the next direction of the inquiry. A wonderings session was held to explore further questions. This led to reassessment and further identification of learning directions. Students worked in groups on a range of learning areas of choice. Teachers engaged in individual, group and class-wide inquiries simultaneously, supporting students in the application of learning across areas, deepening and expanding individual and collective learning in the process. Teachers were facilitators of learning, scaffolding and supporting students dependent on need. Learning partners were involved in the process as partners in out of school excursions, interviews, email, and other forms of connection. The culmination was bringing everyone together for a Night Festival organized by students and learning partners to celebrate and share what they had learned.

A comment was made at the school, that “it was exciting to watch the students get excited, drive their own learning, problem solve and really feel it… This learning experience has been really different—teachers scaffolded, supported, and guided students as they took agency over their learning, made connections to school and the wider community, applied those concepts to a real-world setting, and used digital technology to extend their learning beyond the classroom walls. We believe this type of learning strengthens our students’ understandings of themselves as learners, increases their ability to focus on their own strengths, talents, and interests, and improves learning partnerships between the school and the community.”

Hearing this you can see the practices and outcomes at play in the experience as well as the deeper learning that inquiry enables. The equitable learning partnerships between all stakeholders were leveraged through various assessments where students engaged a variety of partners depending on their learning needs. Students were engaged as active partners in all aspects of the inquiry including the design and implementation of layered assessments that facilitated the measurement of deeper learning outcomes. As well, the students were engaged in a range of authentic, sometimes virtual learning environments that connected them to learning partners within and beyond the community. They took ownership of the direction of their learning and explored meaningful ways they could make a difference with their new learning.

Digital tools were leveraged to connect with partners at different stages of the inquiry and to deepen learning experiences beyond classroom walls. The outcomes were the deepening of students self-understanding, the development of intended knowledge and competencies, and the formation of powerful and lasting connections. This work formed learning partnerships and communicated the learning in the direct development of cultural knowledge, key learning competencies, cross-generational and cross-cultural connections, and understanding of self and others—it was deep learning.

The school culture enabled this work. We have talked about developing a culture of learning, belonging, and high expectations for all, and this work reflected that belonging runs deeper than knowing and understanding our differences—it’s about celebrating them and marveling at their power when paired with another’s. A culture of learning involves the freedom to pursue our interests, share our learning, and improve upon our successes and challenges. It’s student driven and embraces mistakes and failures as opportunities to deepen learning.

As evidenced in the Learning Progressions in Appendix D, who we are as individuals has much to do with others. Along with what we know and can do, who we are involves how we connect with the world, both it’s people and environments. Supporting students to take risks and directing their learning to a range of powerful learning environments connects them to a wealth of knowledge and information, to new opportunities for developing and demonstrating deeper learning outcomes.

The learning experiences described in this chapter are assessment narratives—they evidence learning that’s happening globally and the practices and capabilities that enable it anywhere. The evidence has the greatest impact when it’s measured. At Rapaura school, the shift to deeper learning started with conversations. That initial assessment of current levels of learning led to the identification of a learning direction and then a whole-school and community celebration to kick off learners’ exploration of cultures and festivals. These events are valuable for assessment—there’s a lot to gather from students’ interactions with people and in varied environments. Students then self-assessed their learning, then had a wonderings session to inform the next phase of inquiry, then they took their self-assessment deeper—they chose what their next assessment would be and they designed and implemented it with partners. They collaborated with peers, parents, and community members to make their assessment evidence live in the form of interviews, emails, and observations. Students reflected on learnings from their festival and captured learning in each outcome area. They explored meaningful ways they could make a difference. These are the assessments that tie everything together, asking the question, “How can students evidence and combine their new knowledge, competencies, and self-understanding to connect with others and contribute back?” Assessment is what we DO. The assessments are opportunities enough for informing the measurement of learning and practice.

The Depth in Practice exercise on page 177 can help you begin to align some of your current practice with these concepts and examples. You will familiarize yourself with the application of the tools by undertaking these work examples.

A goal is for students to internalize the curriculum so that it’s with them all the time, which can open the door to explicit curricular learning anywhere, anytime. Deeper learning forms a fluid connection among school, learning, and life. Embedding this in the culture of our schools supports learning to happen at all times now and in the future. It’s important to embed deeper learning outcomes at the onset of your learning design so it’s explicitly embedded. Making these connections is possible if you are intentional in the design phase.

When thinking about student self-assessment and self-measurement, it helps to begin with the evidence. While teachers are familiar with the importance of evidence, students will need guidance to come to terms with it. For example, it was found at Derrimut Primary School in Australia that students often made claims with little evidence to support them, struggled to articulate their reasoning clearly, and needed structured opportunities to support their claims with evidence.

The process for measuring students’ deeper learning outcomes is the same process they’ll use to take actions and make decisions for the rest of their lives—it’s an important skill to have. You now have the frameworks and tools to make this measurement a reality, and you know that this isn’t the teachers’ responsibility alone. It’s important that students develop that same capacity for measurement, designing assessments that foster deep learning, drawing from a wide range of evidence, and matching that evidence to a learning description or Progression to progress as a deep learner.

They have to gather evidence and then use it to determine their own progress and what areas they need to develop. The engagement of the students at Derrimut in research, experimentation, and reflection saw them involved in cocreating assessments and gathering assessment evidence that would support the measurement of their knowledge and competencies.

After you’ve engaged learning partners around what really matters for your learners and identified the outcomes you want to develop and measure, share them with your learners. They need to know the outcomes they’re working to develop along with the practices that will develop them. In the same way that you can’t assess learners if they don’t know the criteria for success, you can’t assess their learning outcomes if they don’t know what they need to do to be successful. Fortunately, with deeper learning, current and future success is one and the same. The learning practice and the partnership are both aspects that can be celebrated.

The Depth in Practice exercise on page 181 asks you to discuss the value of explicitly and intentionally involving students in each inquiry process. For some teachers, this will be the first time they have considered that the students are real partners, and so it’s a true and valuable learning exercise.

The connection between practice, knowledge, and each of the other deeper learning outcomes is an important one, especially given the traditional focus on knowledge.  You already had what students need to know—now you have the practices that will support them to learn in a way that also develops their self-understanding, connection, and competencies. It’s possible within any school or system to achieve this.

Pages 182-188 explore the stories of four schools—one in the Netherlands, one in the USA, and two in New Zealand. In these narratives, the schools develop deeper learning outcomes in literacy, maths, science and technology. These skills were founded in all cases on the values of each school and on an inquiry approach, developing authentic assessments and real, equitable partnerships with students, families, and community members. Activities 9.4, 9.5, 9.6, and 9.7 ask you to use the tools provided to reflect on the capabilities evidenced in these exemplars, to refer to the Connection progression as an analysis tool, and to determine the way the Learning Experience Rubric can be used to familiarize yourself with using it as a measurement tool in relation to the outcomes at Wanaka Primary School. Activity 9.6 specifically guides you to an assessment task using the collaboration tool, and 9.7 asks you to enter the world of using the Learning Experience Rubric to gather evidence to measure your own inquiry approaches. Learning to use these tools will help them become real for you—they will come alive in your practice.

Every detail of the learning experiences in this chapter describe an opportunity for assessment of deeper learning. Once your measures provide an intentional application for gathered evidence of the outcomes that matter, it’s then a matter of matching evidence to measure. The cases in this chapter have similar elements, but they demonstrate something unique about learning and together they give life to the concepts, frameworks, and processes that this work is about. Collectively, they show us what deeper learning outcomes, system capabilities and elements of authentic practice look like when they come together in those deeper learning moments between students and their learning partners.

Deeper learning has a characteristic flow. It starts with a spark from the recognition of something important and relevant in the curriculum. From there, initial assessment reveals current levels of learning and an idea of what will be important to discover along the way, making the necessary curriculum connections. Inquiry guides the course of the experience, branching to and fro as learners’ curiosity and creativity take hold. Explicit attention is paid to the students’ development of deeper learning outcomes and students are active partners in assessment and each other element of inquiry. Students become teachers—they leverage partnerships, environments, inquiry, and technology, contributing to the learning of others, understanding that learning is an active and dynamic partnership between themselves and their learning partners. Assessment evidence informs the measurement of deeper learning outcomes, which in turn informs how the learning moves next. Reflection and change pervade the experience.

What matters in each assessment is that it’s purposeful, it’s connected to the learners, and that the system of assessments collectively work toward every outcome, every day, not just as a summative exercise at the end of a unit of work.

Then, how do we know? How do we know that the practices embedded in the experiences develop deeper learning outcomes? We measure them collaboratively, reflecting and discussing them in a process called collaborative moderation. It’s the subject of the next chapter, and it spreads and shares this practice throughout our schools and systems.

Chapter 10

Collaborative Moderation is used as the process to assess and measure deep learning outcomes and practices.

Chapter 10 – Collaborative Moderation

Measuring and Sharing Learning and Practice

Common moderation processes involve grading or scoring students’ performance on a particular assessment, using rubrics or a scoring guide. It engages a group of teachers within and or between schools, and its purpose is to ensure consistency in grades for individual assessments. It’s important professionally for a number of reasons, not least of which is the value of directly connecting teachers’ professional learning with the examination of students’ learning.

Our moderation process is slightly different. We’re interested in measurement consistency in relation to authentic mixed-method assessment. We focus on inter-rater reliability in relation to Learning Progression and other rubric ratings, made using a range of AMMA evidence. This makes sense in that it takes the focus off individual assessments and places it on how the breadth of assessment evidence comes together to evidence intended learning. It also involves measuring the effectiveness of instructional practice within a deeper learning experience in terms of developing deeper learning outcomes. Student performance remains paramount, and this work takes your own learning one step further in the examination of why students’ learning actually occurred. It requires fully evidenced descriptions of the learning that occurred and how it was enabled or enhanced by authentic instructional practice.

We refer to those descriptions as exemplars, and they take a number of forms—such as websites, slideshows, collections of documents, etc. What’s most important is that the descriptions they provide are enough to fully evidence the learning that occurred, emphasizing what it looked like and why it happened. Exemplars, like the learning they describe, are a celebration.

Exemplar moderation is intended to:

  • Develop a shared language and understanding around deeper learning measures, concepts, and outcomes by engaging teachers and other professionals around the sharing of practice and learning,
  • Collect and identify the practices that develop deeper learning outcomes for application in learning partners’ daily practice and experiences,
  • Build understanding of the current depth of practice and learning, both school and system wide, supporting the identification of strengths and areas for improvement, and
  • Establish inter rater reliability in learning partners’ use of measures for deeper learning.

Deeper learning is strongest when it’s systemic, strengthening capabilities and practices that will enable greater depth. One of the best vehicles for this work is the change team process. Their meetings offer ideal opportunities for small-scale moderations in addition to any scheduled moderation sessions that engage the wider school community.

Both are valuable—ongoing, regular moderation in change team meetings ensures that you are developing, spreading, and acting on best practices in real time, as well as providing professional learning opportunities that engage everyone involved.

Begin with inquiry. Select one focal student for whom to evidence learning and related practice in the exemplar. Gather the evidence required to demonstrate the learning that’s occurring and why.

Describe. Teachers design a learning exemplar that shows that intended outcomes developed and why. Think about what these exemplars might look like in light of AMMA—they should include a range of indicators that combine to fully evidence teachers’ decisions with regard to levels of learning. It helps to build out the exemplar as the learning progresses to ensure that the necessary evidence is captured for sharing. Plus, the learners can be involved in describing their learning as well, and these can be included in the exemplar. Evidencing learning is learning in its own right.

Moderate. Once the exemplars are completed, you’re ready to moderate them with the change team or other schools. Make the exemplars available to all participants. The group should include teachers, principals, and other school leaders and any staff whose expertise will add to the learning experience. Both individually and collaboratively, moderation participants should examine each exemplar, reflect on and discuss the evidence provided, measure learning using the Learning Progressions, and measure the effectiveness of embedded practices using the Learning Experience Rubric. This may last a couple of hours or may be spread across a day depending on the number of exemplars and participants. There’s always efficacy in going slow to go deeper. Doing so ensures your intended outcomes will become a reality and that the process will drive positive changes and deepen everyone’s learning.

Change. It seems a small element of inquiry, but it makes the difference. It’s the reason inquiry is continuous and it’s the source of its transformative power. Changes in practice are integral to the learning process and to deeper learning overall. What you learn from moderation has to be put back into the school or system to focus your efforts on developing the outcomes and capabilities in need of improvement, as well as the practices that will bring them to life.

The Learning Experience Rubric measures the depth of practice embedded in each experience. It’s what teachers and learning partners use to measure their individual experiences, and it’s a tool for collective use in moderation. When thinking about new practices to implement and measure, the practices you read about in chapter 9 are a good place to start. You have the tools, language, and frames to bring these practices to life. They will emerge all the time and you will identify them and bring them to life and then share them. Discover and embed them in the learning process, evidence them in learning exemplars, and spread them through collaborative moderation.

Moderation will tell you a lot about where you need to be in the ongoing change process as well as what you need from each frame of measurement—engagement, development, clarity, inquiry, and depth. As an example, let’s say that your school’s moderation process has identified challenges related to the engagement of parents as real partners.

You’ve identified this as your top instructional priority and you are ready to design solutions. Your professional learning strategy will involve working with parents to jointly understand learners’ interests and needs and supporting them to partner in every aspect of the inquiry process. You’ll work on supporting, valuing and utilizing insights gathered and fostering parent leaders. Throughout this learning process your focus will be on facilitating deeper learning experiences and teaching for 100% success, and gathering evidence in the school and community by conducting assessment that provides a complete picture of parent engagement and its impact on student learning, as monitored and measured using the Learning Progressions. By the time you and your learning partners gather to moderate these exemplars and identify best practice, you’ll have new evidence and learning to share as well as new priorities and professional learning focuses to keep your work in motion.

Each of these elements represents a dimension of the Capability Rubrics which work together to support your breadth of activity. This narrative makes it easy to see how moderation brings everything together. All of your work will be accelerated using the moderation process.

The strength of the moderation, and then its impact, rely on the integrity of the evidence. In the same way that a range of AMMA evidence is required to measure students’ development of deeper learning outcomes, a range of evidence is required to demonstrate both that and why the development occurred. It’s been one of the main questions and considerations to come out of the moderation experiences we’ve witnessed. I’ll outline what an exemplar has to include to allow for fully informed measurement:

Context – Why did you focus on evidencing learning with this particular learner? What curriculum areas are you working in and what are the explicit curriculum links? What outcomes and elements of authentic practice are you focusing on, respectively? This information will anchor the evidence.

Ratings – Provide progression ratings for the student’s development of the outcomes or individual dimensions of the outcomes you’re focusing on. Remember, it’s ok to focus on a single outcome or a single dimension. Doing so develops your inquiry practice. Providing ratings isn’t enough—the main objective of any exemplar should be to provide evidence of why those ratings were made.

Evidence – Evidence of practice and outcomes makes up the meat of any effective exemplar, and the two are linked. What practices are developing what outcomes, and to what extent? You’ve already mapped assessment evidence against the Learning Progressions to measure learners’ outcomes, so that same evidence—whether it’s full or partial classwork, conversations with students or parents, observations, test scores, survey results, or any other AMMA evidence—should be displayed or described in the exemplar alongside your ratings. The goal is for the moderators to be able to use the evidence provided to come to the same conclusions about student learning. You have to provide the evidence to demonstrate that particular practices or a combination of practices led to particular outcomes. The Authentic Inquiry Guides are valuable evidence sources for this work.

What did you do, how did you do it, and why did it work? That’s what’s worth sharing.

Reflection – Your own reflections at all stages of the journey are conversation starters and points for discussion. What were the successes, what changes are needed, and what challenges did you meet? When describing challenges, focus on what was needed or what will be needed to take learning deeper.

The evidence must anchor you at all  times. In all elements of any discussion, refer to the evidence. If it’s not available, constructively share that feedback with the learning partners so that the necessary evidence can be gathered. Remember that moderation is a learning experience, not a high-stakes accountability practice. Its purpose is improvement for all.

It takes time to be assessment literate, to figure out what evidence to include, let alone how best to format, structure, and describe it in exemplars. It pays to start small—focusing on a single learner lessens the challenges and builds capacity until it becomes “what you do.” Some schools or systems develop a template for exemplar design which can include a recommended format along with prompts or questions to get learning partners focused. They should be designed in the way that best demonstrates the learning that occurred. However, learning partners should be encouraged to get creative in how they showcase and celebrate their learning.

What really matters is the capacity to clearly and successfully link evidence of deeper learning with evidence of the practices that bring it to life. Because all tools are utilized in the moderation process, it increases your capacity to use any of the measures consistently. This is inter rater reliability, and it must be established in order for any measure to be valid. If the measures work as they should and are used as they should be, then learning partners should arrive at the same measurement decisions.

Some schools are concerned that this work takes away from instruction time. However, by engaging in moderation of exemplars you can establish inter-rater reliability in the use of measures, further develop your capacity for instruction, and provide valuable evidence to your school or system about what’s working and what support is required.

To continue learning between moderations, it’s helpful to set up and add to an online exemplar bank through which teachers and other learning partners can access moderated exemplars. Similar assessment banks are useful—see the website www.performanceassessmentresourcebank.org.

The exemplars that are designed with the intentions we have outlined have not just the assessment design but the descriptions and evidence of practice as well. These AMMA banks are truly helpful to develop authentic practice. On page 200, you can read about the way schools in New Zealand moderate across a group of schools, and it outlines the way this process increases collective capacity.

The notion of thinking backwards is helpful as you’re planning to engage with deeper learning. You know what you’re working toward, and you have what you need to get there. Look at the exemplar moderation tool in Chapter 8 when setting up and facilitating moderation in your own Change Team, and use the learning you’ve added along the way to think backward, engage deeply with what’s been presented, and connect the dots from where you want to be back to where you are now. It’s a journey, and that’s deeper learning—an ongoing process of learning, depth, and discovery.

Have any questions? Please get in touch!

CONTACT US

Email general inquiries to info@thelearnerfirst.com or find more contact info here.

We have businesses registered in the United States, Australia, and New Zealand and work with educators and schools in countries around the world.

About | Contact | FAQ | Terms

DISCLAIMER

The information contained in this website is for general information purposes only. The information is provided by The Learner First, and while we endeavor to keep the information up to date and correct, we make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the website or the information, products, services, or related graphics contained on the website for any purpose. Any reliance you place on such information is therefore strictly at your own risk.

© 2018 Enfold by Kriesi