Return to my website.

Sunday, November 27, 2016

Addressing PBLA Challenges

I've just returned from attending one day of the TESL Ontario Conference where I made new friends, saw old friends, gained knowledge, handed out a few of my cool new MOO cards, and did A LOT of listening, thinking, listening, some talking, and more listening to others' questions and concerns regarding the roll-out of Portfolio Based Language Assessment (PBLA).
Kelly with John Sivell
I was deeply honoured to be one of the two 2016 recipients of the Sparks of Excellence Award. My morning students emailed me congratulatory notes. My partner and I stopped in Chinatown Saturday to buy a load of treats. I'll bring in some red tea Monday to go with the big white box of red bean paste buns and such, and we'll hang up my framed certificate next to my other award certificate.

Okay, let's move on. The task before us is so large and multi-faceted that I think a good place to start would be to mark off what I--with my little blog--can tackle and what I should not try to take on. The many blog and Twitter comments, emails and private messages I received after writing this blog post can be divided into roughly two groups: those concerning the validity of PBLA itself (e.g., Was enough rigorous research done before decision to adopt? Will there be studies on ROI?), and those concerning practical challenges of implementation.

I have no doubt that PBLA has a lot of value to offer certain classes and something to offer almost all classes. Whether or not it was a good idea to make it mandatory for all IRCC- and MCII- funded programs is a question that will take a long time to sort out. For the foreseeable future, PBLA is not going anywhere. Because I am drawn more to detail work than to big picture work, and because I neither have a degree in education nor am inclined in that direction (applied linguistics floats my boat), I am going to focus on the latter category of concern, though I do hope someone somewhere will continue to pursue the theoretical questions.

Okay, so now that I've narrowed the mandate of what can be tackled in this space, we have to parse that into its component pieces. Within each area of concern, we have to be willing to talk openly and honestly about the troubles some teachers are facing. If we do not do so, we are only going to compound the problem and possibly lose some more very good teachers. (Some have already quit.)

Here are a few areas of difficulty that I was able to identify while listening to presenters and fellow attendees during my admittedly brief one day of participation at the conference:
  1. Many classroom instructors have one set of ideas regarding what is expected of them while workshop presenters and project leads respond to certain questions by shaking their heads in awe, mouths wide open, on hearing what teachers have been told is required of them. (This could be good news; you may be knocking yourself out to do something that is not a requirement of the funders.)
  2. Many teachers feel that their concerns are not truly being heard or are being dismissed or met with a set of scripted comebacks intended to keep them in line, treating them as "resisters" to be conquered instead of gifted educators trying to offer valuable feedback regarding what is actually taking place on the front lines. E.g., can you please stop reframing all our concerns as "growing pains?" It's dismissive. 
  3. Some at the top of the chain seem out of touch and are making assumptions about the teachers having the most difficulty. This includes not having a realistic picture of settlement English classrooms, which are constantly shapeshifting as immigration patterns and refugee-producing conflict around the globe shift and change.
  4. Some (not all) at the top are glossing over the very real problem of the paucity of materials and supports, including pay and benefits, required for the fair and realistic implementation of PBLA.
  5. Feel free to add to this list by commenting below. (It is possible to comment anonymously; not even I will be able to see your real name or email address if you do not give them.)
So with a few of the areas of breakdown having been identified, let's see if we can brainstorm some solutions or at least places to start looking for solutions to these problems.

PROBLEM ONE: CLARITY OF EXPECTATIONS / REQUIREMENTS

It is meaningless for us to debate about the pros and cons of PBLA if we are not even all on the same page regarding what it means, what it entails, and what it does not entail. I heard more than one project lead say that PBLA is meant to be more flexible than many attendees indicate has been communicated to them by their lead teachers. I am very literal-minded. Please tell me where I can go to see which elements are flexible and which are non-negotiable.

1a) collection of artefacts
I watched as teachers began debating back and forth over how many artefacts must be collected per classroom hour for each skill per term. Does this vary depending on the funder (IRCC or MCII)? Of those artefacts, what portion of them, if any, can be skill building activities and how many can be skill using activities? 

In order that we all clearly understand what the funders' requirements are for federally and provincially funded programs, I suggest we create (if there is not already) a central location--such as a wiki or a spot on the new and now usable Tutela 3.0--where all requirements are clearly delineated, perhaps in table format.

1b) choice of assessment tool(s)
Some teachers, myself included, were under the impression we had to develop rubrics. That got us into a discussion of the definition of a rubric versus a checklist or other type of assessment tool. A true rubric that includes a holistic section and analytic section with criteria copied straight out of the CLB tables with (possibly weighted) scoring is not something you can whip up in five minutes. It's time consuming. I had been using rubrics for my end-of-term exit tests before the advent of PBLA. I created them in MS Word based on good examples found online and used criteria lifted straight out of the Canadian Language Benchmarks document. I subsequently attended a TESL Ontario conference workshop a few years back entitled "Happiness is a Good Rubric," which I loved. In that workshop I learned how to simplify my rubrics. (Can I still call them rubrics after I greatly simplify them and go from numeric scores to checkmarks, or are they now checklists even though they still have multiple columns for degrees of achievement?) Instead of needing a second page in order to fit in the holistic section, I started using a final row in my table for an overall "the purpose of the task was achieved" criterion that receives double the weight of the analytical rows. That's my simplified holistic section.

I can create an assessment tool using my intuition and common sense, written in language my students can understand, in five or ten minutes. But that begs the question of accountability and alignment with the CLBs. I know that I referenced the Canadian Language Benchmarks when designing my assessment tools, but how do my colleagues and superiors know I did so when the assessment tool has been so drastically changed in order to make it meaningful and comprehensible to (low level) learners?


I shudder every time I have to use a full sheet of paper per student per assessment and jump at every opportunity to simplify this process while reducing paper and toner usage. So I would like to see it spelled out somewhere how often we must use actual rubrics. Are they only required for Real World Task (RWT) assessments?

PROBLEM TWO: STONEWALLING THOSE WHO GIVE CONSTRUCTIVE CRITICISM
I notice a lot of teachers coming across very testy right now. Some of us are cranky and not always managing to be polite and diplomatic. If you are a project lead or lead teacher or administrator or member of the funding body or government and you are taken aback by negativity, I ask you to reflect on how you are coming across to us. Are you practicing good active listening? Are you talking at us or are you asking us questions and really listening to our answers rather than planning your next retort while we are speaking? I predict that if you talk to us like the adults we are--some of us with masters degrees and degrees in education--we will respond in kind. When someone tries in good faith to bring up a concern and feels as if she is being stonewalled, a natural reaction is often to become louder and more negative, go around the obstacle, go underground and become passive aggressive, or give up and start looking at those office or overseas job ads.


By the way, props to the presenters I witnessed being empathetic, patiently informative and open to feedback of all types. You will be our partners in finding solutions. I give you a 😃 .

We need a forum where we can voice concerns about how PBLA is being implemented, how it is affecting us personally, and what we can do about the biggest stressors. This forum needs to have an option for anonymity in order to protect the livelihood of those teachers who do not feel they can speak freely without reprisal.

Readers, please offer suggestions. I am thinking about a bulletin board I saw in an organic grocery store once. The proprietors put out index cards, pens and a box with a slot in the top so that customers could anonymously write questions or complaints. The store owner would read these and pin them up the following day with his response below on a separate card. It would be great to have a similar forum for PBLA questions and concerns that might eventually make their way into a FAQ document posted on the new Tutela 3.0. I don't think the initial forum can be hosted by Tutela because of the need to log in as a member. This precludes the option of anonymity.

We may also need DIFFERENT or more balanced surveying. I heard from some teachers that whenever they get a survey asking them about PBLA, it doesn't include enough questions designed to collect feedback about problems. We may need to run our own survey. I don't have an education in survey design, but I am very adept in the use of Survey Monkey. We may want to collaborate in designing our own feedback tool.

PROBLEM THREE: ARE SOME AT THE TOP 'OUT OF TOUCH?'
Never have I felt so unseen as while trying to advocate for teachers who are currently considering leaving this line of work due to the stresses of attempting to fully implement PBLA with two classes of up to 24 students each.

At one point a project lead said to me that the teachers currently feeling overwhelmed must certainly be those who were not following the CLBs before. No. No and no and no again. Just no. When I come to you and say that I know of teachers in my city who are sleep deprived and working themselves down to minimum wage with the extra hours they are putting in trying to fulfill all their obligations under PBLA, I am talking about teachers who are some of the most competent I've ever met--teachers the Canadian Language Benchmarks board could hold up as role models for how it always should have been done.

How can we show the project leads everything that we are dealing with in our classes and how different each cohort is, each class is? Here are just a few factors that influence the degree of stress I might feel when faced with implementing PBLA:

  • Continuous enrolment: thanks to our administrative assistant, new students join only on Mondays, but they can appear at any time during the term, including in the middle of a module. It's hard enough getting them up to speed on the content, much less figuring out how and when they will get their orientation to the binder, fill in "My Story," etc.
  • PTSD: many of our refugees are exhibiting signs of PTSD. We are seeing psychological and emotional regression, which is something that some of us feel completely ill equipped to deal with (there is a reason I chose not to teach high school). I'm referring to constant bickering, hyper-competitiveness, more than usual absences, spacing out, crying, talking back, sexually inappropriate behaviour, bullying, teasing, etc.
  • Undiagnosed learning disabilities and behaviour disorders: I am currently dealing with this to the best of my ability. Try getting twelve CLB 1L or Foundations learners to file an artefact in their binders in the right place and copy the title and date onto their inventory sheets while a behaviour disorder is in full flare-up.
I'm sure there are many more points you could add to these. Please do.

PROBLEM FOUR: INSUFFICIENT SUPPORTS
To the funders, administrators and project leads, I want to suggest that you read "Switch: How to Change Things When Change is Hard" in order to get an idea of where you are falling down in the manner in which PBLA is being imposed. It's been a long time since I read it, but the chapter that stays in my memory is called "Shape the Path." If you want a group of people to change their behaviour or adopt new practices, you have a much better chance of success if you make the new way easier than the old way. An example of this is placing the waste bin next to the washroom door if you don't want germ-conscious people throwing the paper towel on the floor.

Rolling out PBLA before providing all the tools is asking for failure. Some of the teachers John talked to said something to the effect of, "I don't want to attend another damned webinar on how to do this, not even if you pay me for that time and certainly not if it's my own time. I want some in-class help, a warm body."

Some teachers will disagree with me on this, and I welcome that dialogue, but I for one would love to have a quality text to use with each level. I liked the old LINC Classroom Activities for 1-4 and 5+, as did my students, but there were not enough of them. That being said, it's the tired me that reaches for off-the-shelf content. When I'm well rested and have extra time, I produce content that is tailor made for the exact needs my group has communicated to me that week. I think I'm a better teacher when I'm giving a lesson that is free to expand or contract, twist or turn organically by the day and by the hour according to what happens in the classroom. It's almost alchemical. It's the reason teaching is an art and not a science. It's the reason some of us feel this field is a vocation, a calling, and not just a way to pay the bills.

I realize not everyone--especially not those who question the validity of PBLA itself--will see this as an answer, but I know teachers who are THRILLED that Conestoga College is working on a bank of ready-made rubric templates. And Rana Ashkar is piloting some multi-level module plans with ready-to-use assessment tasks and tools as well as lesson ideas. They are building a bank of multi-level modules!

So basically what I'm saying is this: there are only so many hours and minutes in a day / week. If I'm to come up with all the content of my courses, where am I to find the time to also come up with all the assessment tools while also doing more marking?

On a related note, if your hourly wage is supposed to include your prep time, do you know how many minutes per classroom hour you're being paid? That is something I personally would like to know. What is the industry standard in Ontario? In other provinces? Are you unionized? Do you think being in a union or not has an effect on how PBLA is being instituted at your agency?

Whew! Okay. That's enough for today.

Thank you for sticking with me through this enormous blog post. I hope you'll leave your comments below, with or without your name. If you comment anonymously, perhaps give yourself a nickname by which we can all refer to you. For example, "Drowning in Durham" or "New Teacher in a Prairie Province."


*As always, please alert me to typos and grammatical errors. The dancing back and forth between American and British spelling (analyze, practise, etc.) is the fault of my American upbringing and the fact that spellcheck for Blogger hasn't learnt yet that I wish to be Canadian through and through.

20 comments:

  1. I love that Conestoga is working on a library of resources! My alma mater making me proud!
    I think that if we're able to use checklists that would be amazing for literacy levels. I think saving time is crucial for us to be happy rested alchemist teachers.

    ReplyDelete
    Replies
    1. Thank you for taking the time to comment, Jocelyn! I agree that saving time is crucial. I have decided to start keeping track of the time I put in outside of class hours and set a limit for myself. I want to get back to some of my hobbies and do better self-care (more sleep, meditation time, birding, sewing).

      Delete
  2. TESL Windsor just posted this on their FB page:
    There were a few misconceptions about PBLA that we heard at the TESL ON conference that we were able to clear up:
    * According to research, it takes about 300 hours of instruction to move a benchmark level. So AT THAT TIME, i.e. around 300 hours, Ts want to have sufficient evidence in a portfolio to evaluate progress, i.e. 8-10 artefacts (a combination of assessment and SU activities).
    * 300 hours of instruction is 15 weeks @ 20 hours a week. This means 2.6 or 2-3 artefacts (assessment tasks and/or SU actvities) per week.
    * 300 hours of instruction is 50 weeks @ 6 hours a week. This means 1.6 or 1-2 artefacts every 2 weeks (assessment tasks and SU activities).
    * Assessment tasks need to be accompanied by an assessment tool. This tool does not HAVE to be a rubric. It could be a rubric, a checklist or a rating scale. N.B checklists and rating scales are easier to make than a good rubric, and MAY be easier for CLB 1-3 Ss and ESL Literacy Ss to understand.
    * We reminded Ts that altho there are some 'non-negotiables' or 'must-haves' in PBLA, e.g. NAs, goal statements, learning reflections, etc. WHAT THOSE ITEMS LOOK LIKE IN YOUR CLASS IS UP TO YOU. For example, NAs might (should?) be pictorial in ESL Literacy and CLB 1-3, but might be a survey or questionnaire or a synopsis of a brainstorm discussion, etc at other levels.
    * A NA where Ss prioritize from a range of social contexts & tasks might be carried out at the beginning of a term or the NA might be a choice between 2 topics or 3-4 topics held right before a new module, etc.
    * Apparently, someone on Twitter said Joseph of MCI said Ts didn't need 8-10 artefacts in their portfolios. We are assured that he did NOT say this. MCI and IRCC continue to stress that Ts are STRIVING FOR 8-10 artefacts in their portfolios to make a CLB decision to ensure sufficient evidence. In early days of PBLA, Ts might not quite get there but it's a goal they should continue to work towards.

    Hope this helps!

    ReplyDelete
  3. I am "someone on twitter" who sat in on not one, but BOTH of the MCIIT Q and Q sessions with Iveson, Murad and Joseph. My tweet reported that Joseph said 8-10 WERE NOT NECESSARY. He emphasized that what the intention was, was to create regular, experiential assessment planned with each unit or module, and that if fewer artefacts were obtained, but that the teacher felt that the student ought to progress, and had the evidence to back up that claim (even if fewer than 8 artefacts), then that student would be able to progress.

    Let me repeat - I was in BOTH sessions. Joseph from MCI actually brought up the 8-10 artefact count HIMSELF (capital letters are used here for emphasis and do not in any way mean that I am shouting at you). He told an entire room that the 8-10 count was the most frequently brought-up concern from instructors.

    Now, perhaps I am losing my mind and I misheard him not once but twice. I plan to forward this on to several of my colleagues who were also in the room. If I am wrong, I am more than willing to take to twitter to correct it. Joseph very generously gave out a contact email as well, so I might as well get it straight from the horse's mouth. I will be back to comment again shortly.

    Thanks for this blog - this conversation is needed.
    Jen Artan

    ReplyDelete
    Replies
    1. Thank you, Jen. I think that if Joseph said this, he is a wise man. I have a new student from South Sudan who was doing no work her first week; she sat scowling and looking shell shocked. Week two she achieved CLB 1 level work. Two weeks later she is doing CLB 1 (NOT CLB 1L) level work easily and quickly, getting bored out of her skull with all the repetition that the other students require, and is demonstrating R and W skills bordering on CLB 2. I am not going to hold her back in January no matter how many artefacts I have. Maybe the proper action is to call the Assessment Centre to let them know she was wrongly placed. Whatever. We know our students and we know what we know.
      Thank you for your kind words. --Kelly

      Delete
    2. I was at the first session and am sure I heard the same re the 8-10 artefacts.

      Delete
    3. Thanks, Elizabeth - I appreciate that. I've asked a few colleagues as well. They've agreed that they heard the same message. Hoping they'll post that themselves, in here.

      Delete
  4. I am very surprised at hearing the concerns of so many teachers with PBLA. I have no real problems with it , perhaps because at my board we were doing task based learning for years before this. We have classes of between 30 & 40 students with continuous enrollment, and this is my 2nd term of using PBLA and I find it helpful. There is so much already out there-lessons, modules, assessments, you need to create almost nothing. Perhaps more instructions on how to find things to use are what's needed. I hate seeing people stress about this when so much is available to help

    ReplyDelete
    Replies
    1. Thank you for that input, Stephanie! My colleagues and I have also been doing TBL all along. In the interest of ensuring we are comparing apples with apples, may I ask what level you teach? What is the demographic makeup of your class (Mostly refugees or immigrants? What parts of the world are they from? Age group? And how much paid prep time do you have? Thank you!

      Delete
  5. I'm sure seeing a lot of griping in these blogs and comments. I think that ESL is better off with the standardization that we have now. And as for the "heavy binders"... what did students have before? They had their own binders that were just as full of paper as the LC binder. Weeding out no longer needed sheets is a help in this regard and has to be done weekly or at least after a topic has been covered and an assessment given.
    Let's try to be a little more positive and accept change as a necessary evolution of ESL teaching!

    ReplyDelete
    Replies
    1. Where do you see "standardization"? As one instructor told me "you can't talk about doing PBLA to anyone from a different location because everyone is doing sonething different' . (Maybe one reason is because those that designed this exoeriment made erroneous assumptions about classes, learners, instructors/teachers across Canada.)
      Your question " .. what did students have before?" They had CHOICE! They are adults. They could have whatever binder they wanted - zipped, soft cover, fold out etc - OR - the plastic bag binder!! Or no binder!
      Of course I think students should be organised in order to make learning easier and effective and I thought at first providing each student with a binder was a generous (taxpaper) gesture - until I saw,and held the binders and saw the content. They are not "textbooks", nor calibrated to levels - cmon Information officers and CCLB - what were you thinking? That it looks good giving students Canadian information ( that gets dated quickly and is all available online?) Is this part of trying to prove that CLB is better than CEFR?
      Your "heavy binders".. hmm...have you weighed them? A colleague did. Interesting results: The "old" literacy binder - 1.51 kg ( with 394 g artefacts and dividers) The "newer" - 1.014 (remaining dividers, no artefacts, no class work). There seems to have been a reduction in weight. Cost issues? Complaints? LC CLB 1-4 - 1.07kg and LC 5-8 1.21 kg. These are not classroom work binders - so some students have TWO binders now. Apart from the fact thst the focus is now on artefact collection instead of language learning...
      You think PBLA is "necessary evolution"? What proof do you have that this is positive "evolution" and not regression?


      Delete
    2. Hey Anonymous, I really hope you read this! How would you like getting paid less than minimum wage for all of the work you do and get told it's not up to some mythical standards no one can put on paper as standardization. Please, please show up at my school and I will happily show you students who are frustrated by assessments and more assessments to fill the binder to appease the"standards" which not one lead teacher or PBLA guidebook shows.

      Delete
  6. Dear Anonymous Supporter of PBLA,

    Thank you for your interest in the needs of the TESL instructors that are reaching a critical mass of frustration over trying to implement a process that is not executable within the confines of their work contracts. I see that you disregard the expressions of inability to complete the task. I wonder how learners would react if addressed in such a manner in the classroom. "Class, you have to do a 5 hour presentation tomorrow. You will negotiate all aspects of your presentation and manage all support and resource needs. Everything that I taught you over the last 5 months is no longer valid or applicable. The support texts that were created by TDSC TDCSB and other ESL schools are not valid guides. You need to ask people who have not been trained to run a classroom for advice on how they want you to do the presentation. Oh, there are 30 of them and they all need to be pleased. Or they will report dissatisfaction on their self assessment forms which will go into their LC which will be seen by others. Of course you can do this as you are a professional student. There are no created materials or resources, and supplies need to be sourced for realia. Don't worry though as you have 5 minutes per hour of preparation time. That surely is doable. Any questions? Sorry, I will not listen to your griping. No, I don't have any answers for you. Just do it. The "funder wants it"..." And you will do this 4 times every two weeks. That's right, all the research, resource creation and materials resouring for 4 major presentations every two weeks but did I mention all the skill building and skill using activities that you need to create leading up to the presentation. IMPOSSIBLE, now you are griping. Be more professional. Really, I am disappointed in you"


    What would happen if we dismissed learners as instructors are being dismissed? Surely help is needed. Too many people are exhibiting stress reactions. Too many employers are taking the easy street and saying "The funders want it". Too many SPOs are watching the slow disintegration of their staff. Could you be a little more positive and fight to find a balance as change is a reality life but PBLA is not a necessary evolution of ESL teaching. PBLA is a process that is being independently implemented across a country without a national monthly "anonymous" employee survey to capture the crisis as it has unfolded. Have you had your head in the sand over the last several years? Where is PBLA not causing problems? Changes in the automotive factory where my husband works are implemented better and the workers day is balanced when new chores are tasked to them. Education workers should have it atleast as good.


    No one here has said that they do not want to be accountable. Everyone acknowledges that the CLB is their guide. Everyone is pursuing the work. Yet, when people express that something is failing, you make a personal judgement on them. I find this kind of insult a very sad indicator of the division that PBLA has caused. This is the change that I have never seen in ESL before. Cold, uncaring and unfortunate Anonymous should take some time and consider their colleagues, but maybe you aren't an instructor? Maybe you are admin and you don't have to tap dance to the PBLA quartet. Anyway you slice it, there are people who are suffering in fear that if they complain that they will be disciplined. Surely, that is not ever a good "evolution". Do you know that evolution is a theory? I don't like it. But empirical process can be proved and supported so why don't you, Anonymous, look for an empirical process to help others with this PBLA problem and be a little more positive by showing us how you do it or don't. Can we get from this mess to a clean process? Sure, but not without peeling the onion. There is something rotten about PBLA that smells pretty bad.

    ReplyDelete
    Replies
    1. Brilliant. I keep trying to avoid terms like rotten or corrupt but it is time to ask WHO is at the "head" of this? Where did the buck stop? And when is this anonymous person or persons or body (group) going to account for the time, effort, money spent on this?

      Delete
    2. Anonymous,
      Your comment is indeed brilliant. I think it could be a guest blog post in its own right. --K

      Delete
  7. Sadly, Conestoga documents are felt as not valid and incorrect by our lead teacher, and many staff were told to throw out and redo their modules and assessments.

    ReplyDelete
    Replies
    1. Anonymous,
      You bring up a very good point. Just because you use one of the college's templates does not guarantee that you end up with a valid assessment. I am one of those keeners who goes out of her way to attend workshops, buy books, study like hell. And I have read books about assessing. Yet I still feel ill equipped to design assessments that are valid. So many times I realize I've spent an hour or two creating an assessment only to realize that I've failed to properly isolate the skill, for example. Does a listening assessment require the student to read something? If so, you have not successfully isolated the target skill. This is assuming that we should even be teasing out language into separate skills to begin with! I prefer whole language, holistic measures. Can the student do it, no matter the degree to which he/she relied on strategic competence or compensatory tricks?

      The first time I used a Conestoga College template, I still managed to screw it up. I know I did because several students with higher level skills (CLB 5/6/7) were tripped up by one of my (supposedly CLB 4) questions. In a situation like that, I need to admit my failure and toss out that question.

      It is my view that while we may be qualified to use simple checklists, for complex rubrics we should probably be sticking to those found in books like On Target, as they were written by folks with far, far more assessment creation expertise than we LINC teachers have.
      Sigh. What a mess. --K

      Delete
    2. Reply to Failed Conestoga Assessment,

      Why does your lead teacher have the ability to do that? IF the lead teacher has the ability to say your work is wrong, then they should have the ability to correct the work rather than ask that it be binned. Something is really wrong here. Something is very sad. I hope it gets corrected.

      Try to stay focused on the fact that you are doing the best job that you can.

      Delete
  8. Is it time to address these issues(All of the above) collectively to the funders?

    ReplyDelete
    Replies
    1. Hi, Nov. 17, 2017 Anonymous,
      I would love to see that. It’s very difficult to start a petition or such when so many feel they cannot safely sign their names. Ideas?

      Delete