Quality Performance in the Refugee Protection Division 2019-20

​​​​​​​​​Report of Results

Prepared by:

Anna Purkey and Strategic Planning, Accountability and Reporting (SPAR) Directorate

On this page​

• 1.0 Context
• ​ 2.0 Performance Results
• 2.1 Timely and complete pre-proceeding readiness
• 2.2 Respectful proceedings
• 2.3 Focused proceedings
• 2.4 Reasons state conclusions on all determinative issues
• 2.5 Decisions provide findings and analysis necessary to justify conclusions
• 2.6 Reasons are transparent and intelligible
• 3.0 Results for specific populations
• 3.1 SOGIE hearings
• 3.2 Women fearing gender-based violence
• Appendix A - Checklist
• Appendix B - SOGIE quality review checklist

1.0 Context

This report describes the results of the measurement of quality in decision-making in the Refugee Protection Division (RPD).

Sample methodology

The study reviewed 66 out of a possible 1418 claims and applications that were finalized between April 1, 2019 and June 30, 2019 (the assessment period) on their merits after an oral hearing before a single-member panel. The hearings were randomly selected in proportion to region, outcome, and language of proceeding. All hearings with a duration between 45 minutes and 3 hours were included in the final population. Members who had been hearing cases for less than six months from the start of the assessment period were excluded from the sample. Within the sample, hearings regarding specific populations were identified and further analysed. These included hearings related to Sexual Orientation and Gender Identity and Expression (SOGIE) and Women fearing gender-based violence (WFGBV).

The following charts illustrate the sampling makeup, which is proportionately representative of the population data:

Assessment methodology

This qualitative assessment was performed by an independent reviewer who is a lawyer, and legal academic with a long history of engagement with immigration and refugee law. The reviewer examined all evidentiary and administrative materials on file^{Footnote 1}​, listened to the complete audio recordings, and assessed these against qualitative indicators in a checklist developed by the Strategic Planning, Accountability and Reporting (SPAR) Directorate and approved by the Deputy Chairperson of the RPD (see Appendix A). Each indicator is assessed along either a 1-to-3 rating scale or a binary yes-no scale. The checklist assesses thirty indicators across six performance categories:

1. Timely and complete pre-proceeding readiness
2. Fair and respectful proceedings
3. Focused proceedings
4. Reasons state conclusions on all determinative issues
5. Decisions provide findings and analysis necessary to justify conclusions​
   
6. Reasons are transparent and intelligible

Sixteen of the indicators were mandatory for assessment, and fourteen were assessed only when applicable. Each performance indicator is assessed along either a 1-3 rating scale or a categorical yes-no scale.

The 1-3 rating scale is as follows:

1=Does not meet expectations: The quality requirement was not met. The evidence showed one or more key instances where the proceeding or reasons would have markedly benefited had this requirement been met. There may have been an effort to apply the requirement, but the level of achievement fell short of expectations.

2=Meets expectations: This is a level of acceptable achievement. On balance, the member satisfied this quality requirement though there is margin for minor improvement.

3=Exceeds expectations: This is a level of consistent, above-average achievement. The evidence shows a grasp of the quality requirement and an understanding of its importance to a high-quality proceeding or decision, as the case may be.

Results are also expressed as a percentage of hearings that meet expectations, by obtaining a score of 2.0 of higher.

In addition, to support Gender Based Analysis Plus (GBA Plus), a second checklist was developed for hearings involving persons of diverse sexual orientation, gender identity and expression (SOGIE) and is used in quality measurement reviews in all four of IRB’s divisions (see Appendix B).

Considerations/limitations

Results are accurate to within ±10% percent, 18 times out of 20. This margin increases when data is broken down by region or claim type. However, the goal of the study was not to generate statistics but to identify areas of strength, concern, and patterns in decision-making quality.

This study acknowledges the inherent limitations of qualitative research, which does not generate precise data as does quantitative research. To mitigate the inherent limitations of qualitative research, detailed performance indicators were provided to the reviewer to help focus the assessment. To ensure quality and consistency in the assessment, a reviewer was selected based on their in-depth knowledge of the IRPA and refugee matters. Moreover, a small sample size limits the inferences that may be made about the broader caseload. Where sample sizes are too small, observations or recommendations may still have been provided but these are not based on representative findings.

This report aims to provide a perspective to improve the Division’s performance overall. The evaluation unit provided the statistics found in the table accompanying each result section as well as the information in 1.0 “Context”. However, the findings in this report, including all strengths, areas for improvement and recommendations are solely those of the reviewer. Their observations are necessarily subjective in nature and do not lend themselves to firm conclusions on legal matters such as the correct application of the law, the weighing of the evidence, or the fairness of the proceedings from a natural justice perspective. Only a court reviewing the case can arrive at such conclusions.

2.0 Performance results

What was measured

Each performance result in sections 2.1-2.6, contains a table representing the number of hearings assessed for each indicator, the average score, and the percentage of assessed hearings that scored a 2.0 or higher. The average score is a finding that helps determine which indicators had strong or weak results and helps to inform observations and recommendations. The number of hearings assessed are provided for reference and context only.

There are two performance targets for this assessment:

• The first target is for each indicator to achieve a score of 2.0 or higher in at least 75% of all hearings. This was achieved for 29 out of 30 indicators and is evident in the tables provided (see “% of hearings scoring at least 2.0” column in the table below). Where an indicator does not meet this target, it is addressed in the reviewer’s observations following the table (Areas for Improvement).
• The second target is to achieve an overall aggregate score between 2.0 and 3.0. This study calculated an aggregate score of 2.4 out of 3.0 for the Division, meeting the target established by the IRB.

2.1 Timely and complete pre-proceeding readiness

Why measure this

The groundwork for a quality hearing and decision is set when the Registry prepares a timely, organized, and complete case file and the member assimilates the facts and key issues of the case.

Considerations:

All indicators are applied to all hearings. In two hearings the physical files were not available due to COVID-19 related office closures. They were still assessed using only the information available in the electronic record management system and the audio recording of the hearing. Given the absence of the physical file for two hearings, indicators 1,2,3 were not evaluated for those two hearings.

General observations:

• None Identified

Strengths

• File preparation: Overall, the files were delivered to the members in a timely fashion, were complete, included all of the appropriate documents, and were well-organized. The cases that scored below “satisfactory” on these elements generally had relatively minor issues.

  In the great majority of cases, the Hearing Disposition Record (HDR) was affixed to the inside cover of the file and was completed in full.

  One best practice of note is the uniform tabbing system used in the Central region. Every file is marked consistently with color-coded tabs which make it easy to find all of the most important documents quickly.

• Member preparedness: The study found that in 97% of cases the member was adequately prepared for the hearing. This finding is based on a global evaluation of a variety of factors including the member having prepared a list of consolidated documents, and a statement of the issues, and on his/her apparent familiarity with the facts of the case as well as with the documentation submitted and the National Documentation Package (NDP). Another important indicator of preparedness is the adoption of a clear, focused, and organized line of questioning.

Areas for improvement

• In 5 cases the files were delivered to the member fewer than 10 days prior to the hearing. The reasons for late delivery were not investigated but may have been due to factors outside of the IRB’s control. Several files, both late and on time, had clearly been passed between the assigned member and other branches of the IRB several times – likely in response to late disclosures. Late delivery of the file did not have a noticeable impact on member preparedness in the cases assessed but, in several cases resulted in the member having to take a brief break in order to review documents that were submitted at the last moment, thus extending the length of the hearing time.

• File preparation: Although most files were complete and well organized, in at least five cases there were missing documents. For example, in one case, photographs were missing from the file. It is possible that the original photographs were returned to the claimant, however there was no indication to that effect. In several other instances counsel for the claimant was able to provide the missing documents to the member at the hearing, or the documents were deemed not to be essential (as in the case of a missing notice of change of representation).

  Five files (not the same five noted above) also fell short in terms of their organization. In those cases, either documents had been slid into the file without being properly fastened, or documents that should have been in the main body of the file, such as the consolidated list of documents or the transcript of the reasons, were instead included in the communications file at the front. In only one case did the member comment explicitly about the disorganization of the file received.

  One minor discrepancy in terms of the content of the files pertained to the table of contents of the NDP. While most files included the table of contents, not all did.

  Although each region was internally consistent, there were variations in the completion of the file cover between regions. All of the dockets from the Central region noted the final disposition of the case on the file cover, while none of the files from the Eastern or Western regions did. Both the Western and Central regions clearly indicated on the cover when the file was delivered to the member prior to the hearing, while in the Eastern region the final notation prior to the hearing was SPR-SCAN (rather than the member’s name). Moreover, in the Eastern region the format used for the dates on the file cover varied between MM/DD/YYYY and DD/MM/YYYY, thus creating the possibility for confusion.

  While the differences in practice in file preparation are relatively minor, adopting a set of consistent best practices across all regions can help to reaffirm the unity of the Tribunal, as well as ensure that all regions are meeting the same high standards both substantively and procedurally.

• Incomplete Hearing Disposition Record (HDR): As noted above, in most cases, the Hearing Disposition Record was completed in full. However, in at least four out of eleven cases involving SOGIE and five out of thirteen cases involving women fearing gender-based violence, the “Indication of Hearing Results” section was incomplete and contained no mention of the applicability of the relevant Chairperson’s Guidelines, even in cases where the member had referred directly to them in the reasons. In addition to this issue, there were minor discrepancies in some of the other HDRs, including failure to mark whether the decision was positive or negative, and failure to include the name of the interpreter. Overall, however, with the exception of the issue of the Guidelines, the HDRs were well completed.

• Member preparedness: Although only two cases scored a “1” for member preparedness, at least four of the other cases that scored a “2” also clearly left room for improvement on this criterion. A number of reasons form the basis for this scoring including the following:

  1) the member demonstrated a lack of familiarity with the narrative and the facts of the case (for instance, in one case a claimant was asked what her father’s opinion of her situation was when her father’s death was clearly stated as an important factor in the narrative);

  2) the member demonstrated a lack of familiarity with the country conditions and the information contained in the National Documentation Package;

  3) the member’s questioning was halting and punctuated by long pauses during which the member could be heard to be flipping through (and seemed to be reading) the documents contained in the file. In one case, the member did not appear to have decided upon the issues prior to arriving at the hearing, while in another the member blamed a disorganized file for his failure to have reviewed all of the documents submitted by the claimant before the hearing.

  Although momentary memory lapses are perhaps not unreasonable on occasion, member preparedness is a critical factor in setting the tone of the hearing and ensuring that the hearing proceeds smoothly and, above all, fairly.

Recommendation

• Encourage the Eastern and Western offices to adopt the Central region’s method of tabbing files consistently as a means of helping to ensure that the files are both organized and complete.
• Provide guidance to all regions regarding the importance of completing the HDR and the file cover consistently.

2.2 Respectful proceedings

Why measure this:

Individuals appearing before the IRB expect that they will be treated with sensitivity and respect. Any shortcoming in this regard potentially undermines tribunal integrity and public confidence.

Considerations:

Indicators 5 and 6 are applied to all hearings while 7, 8 and 9 are scored on an as-applicable basis.

General observations:

• The majority of members performed their duties with the sensitivity, respect, and professionalism that is expected. Regardless of the final decision, the impact that the member’s tone and manner of questioning can have on the ability of the claimant to properly present his/her claim and, by extension, on the overall fairness and integrity of the refugee status determination system requires a particularly high degree of review.

Strengths

• ​Sensitivity and respect: In 87.9% of cases (n=58), the members treated the participants (the parties, counsel, witnesses, and interpreters) in an adequately sensitive and respectful manner. In these cases, members generally maintained a high degree of professionalism even while exhibiting compassion and respect for the claimants during difficult, and at times emotional, testimony. Specific examples of particularly good practices include the following:
• In one case the member clearly explained at the beginning of the hearing that he would be typing notes throughout the claimant’s testimony but reassured the claimant that he would still be paying attention. This assertion suggests an awareness of, and consideration for, the potential perception or perspective of the claimant.
• In another case the member took time to reassure the claimant that he would listen to everything he (the claimant) had to say, that they were not there just to “get rid of [him] and get home,” that there was no hurry and they had all the time needed.
• In one case, when explaining the importance of breaking the testimony up into short pieces for the interpreter, the member made a point of using inclusive language in the direction (“this is something that we can do,” rather than “you must…”). This approach established a positive tone for the questioning that followed.
  
  

General examples of good practices identified in the reviewed cases include:

• Informing the claimant that he/she can request a break at any time.
• Acknowledging when the claimant becomes clearly overwhelmed and emotional and suggesting on the member’s own initiative that the parties take a break when deemed necessary.
• Explaining the order of questioning and process of the hearing to the claimant in a clear and straightforward manner.
• Adopting a measured, professional, and patient tone and demeanor even in challenging situations.
  ​
  
• ​Presentation of evidence and representations: In 95.5% of cases, parties were given adequate opportunity to present evidence and to respond to issues raised by the member or by the other party. In many of these cases, only the member examined the party, and counsel’s submissions were waived. In cases that scored satisfactorily on this indicator, the claimant was afforded the opportunity to respond in detail to the member’s questions with minimal interruptions. In several cases, the member even provided the claimant with the opportunity to add anything he/she felt had been overlooked or left unsaid at the end of the hearing.

• Questioning and clarifications: in over 90% of the applicable cases, members asked questions of clarification when the evidence was unclear or the claimant’s response inadequate. In several instances the member collaborated with counsel, or on occasion the interpreter, in order to clarify certain points – either re-framing questions or allowing counsel to do so. Of particular note are five cases where the member identified areas where there was a lack of evidence in the record or testimony early on, thus allowing the claimant to adequately address these shortcomings.

• Communication in the absence of a party: In the four cases where communications had clearly occurred during a break or prior to the start of the hearing, the member summarized the communications on the record. In several cases, it appeared that the recording was not turned on immediately (at the beginning of the hearing or after a break) and so the audio recording did not capture the full hearing (in one case there was no recording of the reasons, in two other cases the recording missed some of the questioning). In one of these cases the member very briefly summarized what had been said, while no reference was made to the failure to record the full hearing in the others.

• Interpretation: Interpretation frequently presents challenges during the hearings and it is difficult to evaluate the quality of the interpretation in each case. Nevertheless, insofar as could be assessed, in 93% of the evaluated cases, any issues with the interpretation were identified and properly addressed by the member. In five cases, the member specifically identified problems with the interpretation and intervened. For example, in one case the member intervened when the interpreter failed to provide a full interpretation of the questions asked. In several cases where the claimant spoke some English or French, the member intervened when the claimant failed to wait until the interpreter finished his/her interpretation before responding.

Areas for improvement

• Sensitivity and respect: In eight cases, members scored a “1” for not meeting expectations with regard to the sensitivity and respect owed to the claimant. In these cases, the members were found to be impatient and rude to the claimants without justification, often interrupting or cutting off the claimant repeatedly, and at times adopting a confrontational or aggressive tone in questioning. The members in these cases lacked sensitivity with regard to the impact that their tone and manner had on the claimants, who were often very nervous and distressed, and on their ability to present their claims. In four other cases, which received a score of “2”, members also clearly demonstrated their impatience in a problematic manner. Finally, members in four cases demonstrated a lack of respect in their communications either with counsel, or with the interpreter.

  In most cases, the lack of sensitivity or respect was noted in terms of the adoption of an impatient, skeptical, or annoyed tone, thus changing the nature of statements or questions that might otherwise appear neutral. Examples of this include the following:

  • Interrupting the claimant with comments about the limited time available;
  • Reprimanding the claimant for being “verbose”;
  • Repeatedly interrupting the claimant, counsel or the interpreter in a brusque and disrespectful manner and preventing the claimant from finishing his/her thought
  • Arguing with counsel (in one case, the member argued at length with counsel about whether he should consider an 11-year-old to be a “child”);
  • Making comments about the fact that people in the claimant’s situation often have falsified documents (prior to any questioning and without there being any indication to the effect that that was the situation in the case at hand);
  • Interrupting and correcting the claimant while making inaccurate statements about the narrative;
  • Making statements about claimant’s situation, rather than asking open questions and then asking the claimant “do you agree?” – thereby forcing the claimant into a position where he must frequently contradict the member in order to fully express himself.
    ​
    

  As it is the member’s role to ensure that the hearing proceeds efficiently, it is entirely reasonable for the member to occasionally interrupt the claimant or counsel in order to re-direct the hearing. Nevertheless, the manner and frequency with which these interruptions and interventions occurred in these cases raises concern as to whether the claimants in question were truly able to adequately present their cases in the best manner possible.

  Refugee hearings are frequently emotionally charged for the claimants. Difficulties with language or interpretation and cultural differences may cause frustration for all involved, including the member. Maintaining a high level of professionalism, sensitivity and respect is both challenging and essential to ensuring the integrity of the process particularly in light of the vulnerability of many claimants.

• Ability to present evidence and make representations: As noted above, in 95.5% of cases, claimants were fully able to present their evidence and make the necessary representations. The few cases that were scored “unsatisfactory” were due to the claimant being cut off or repeatedly interrupted by the member.

  In at least two cases where the member appeared to be taking verbatim notes by hand, he/she required the claimant to present his/her evidence a few words at a time and required frequent repetition. In these cases, the claimant’s ability to testify was severely constrained. When a member is taking verbatim notes, they may require the claimant to stop or repeat him/herself repeatedly. Similarly, when a member is typing notes, there are often delays between the claimant’s response and the member’s next question. One member drew attention to these challenges by reassuring the claimant that he was still listening even though he was not looking at the claimant and would be typing throughout. Claimants in these situations are often testifying about challenging and potentially traumatizing events, to have to do so before a member who is writing or typing throughout the testimony and who, as a result, is not making any eye contact or reading the expression of the claimant, likely impacts both the claimant’s ability to testify and the member’s ability to effectively assess the credibility of that testimony.

• Interpretation: Overall, acknowledging the challenges of working with interpretation, most cases met the required standards. Where problems arose, they were frequently beyond the control of the members. For example, in three cases the interpretation was done by phone with the interpreter in a secondary location. Although all parties made the best of these situations, this method of interpretation is clearly far from ideal given technological/sound and comprehension issues that arose. In these cases, the interpreters frequently had to ask the parties, including the member, to repeat themselves. These interruptions were on occasion time consuming and interrupted the flow of the hearing, impacting both those asking the questions and the claimant’s ability to testify effectively.

  One recurring challenge pertained to claimants who spoke some English or French but were not sufficiently fluent to proceed without an interpreter. In those cases, when they understood the question, the claimants had a tendency to respond directly as opposed to speaking through the interpreter. The member would often intervene and remind the claimant to speak through the interpreter, occasionally becoming clearly frustrated. Although these cases were dealt with consistently for the most part, the constant interventions by the member reprimanding the claimant for failing to speak through the interpreter created tension during some hearings.

  Issues with interpretation that were not addressed adequately by the member arose in seven cases. In one case, much of the additional work in terms of ensuring that the claimant understood the question was done by the interpreter, rather than having the member re-phrase the questions. In three cases, issues with the quality of the interpretation were identified by the claimant him/herself (1 case) or by counsel (2 cases). In one case the claimant drew attention to the fact that the interpreter had not interpreted his response completely and had in fact interpreted the complete opposite (did vs did not) of the claimant’s statement. In at least one case the assessment identified a mistake in the interpretation (the claimant clearly says “Texas”, but the interpreter translates it as “another state”). In two cases the claimant either went on at length without a break or spoke over the interpreter, creating confusion and making the interpreter’s job even more challenging. In these cases, an intervention by the member would have been helpful and appropriate.

  Two cases presented particular concerns. In one, the interpreter was late and the member began the hearing without waiting for him. Although in that instance the claimant knew sufficient French to be able to understand, proceeding without an interpreter when one has been requested could raise procedural fairness issues. In the second case, no English-Kinyarwanda interpreter was available so two interpreters were used – one to interpret from English to French, and the other from French to Kinyarwanda, and back again. Using more than one intermediary exponentially increases the potential for misinterpretation and misunderstanding.

Recommendations

• When planning professional development programming, the division should consider the above findings on sensitivity and respect and provide training to members on the impact that their manner of questioning and tone can have on the ability of claimants to testify effectively.
• Whenever possible, avoid the use of interpretation by telephone and especially the use of multiple stage interpretation.
• Have a stenographer prepare a simultaneous transcript of the hearing so that members are not required to take detailed notes during the hearings.

2.3 Focused proceedings

Why measure this:

Proceedings that are efficient and well managed create conditions for quality outcomes to emerge and support the IRB’s efforts to make the most effective use of its resources.

Considerations:

Indicators 10 to 12 are assessed against all hearings while 13 to 19 are assessed on an as-applicable basis.

General observations:

• None identified

Strengths

• Issue agenda: In 81.8% of cases (n = 54) the member set out a clear issue agenda at the beginning of the hearing. In most of these cases, the issue agenda was specific to the facts of the case at hand (for instance, identifying specific cities when the existence of an internal flight alternative is at issue, or identifying the claimant's identity as homosexual as being the issue rather than stating “identity” generally). Additionally, in the majority of cases the member provided a detailed explanation of the plan of questioning; explaining to the claimant the order of questioning, when they would take a break, and the possibility of receiving a decision that same day, etc.

  Members often used this opportunity to provide instructions to the claimants regarding speaking slowly, breaking up their responses to facilitate interpretation, the possibility of requesting a break if needed, etc. This exchange provided a valuable opportunity for the claimant and the member to interact in a positive and constructive manner prior to the start of questioning. In the Western region, members started the hearing with an issue agenda 100% of the time.

• Member's questioning - relevance: In nearly 85% of hearings, the member's questioning and the claimant's testimony and documentary evidence were relevant to and focused on the issues identified in the claim.
• Member's questioning – focus and organization: In nearly 90% of cases, the member's questioning was focused and well-organized. Best practices in this regard included following the issue agenda set out at the outset, proceeding in chronological order when questioning, asking open-ended question and requesting follow up or further explanation when needed, and allowing the claimant the latitude to respond with minimal interruptions other than those necessary to maintain the focus of the testimony.    
• Hearing completion: All but one of the hearings reviewed finished within the time allocated for them, with the average hearing duration being 84 minutes. 
• Management of challenging situations: Overall, members did an excellent job of managing challenging situations including problems with videoconferencing technology, a fire alarm test, assisting self-represented claimants, and managing distressed and emotional claimants. In one case in particular, at the very end of a highly emotional hearing, the claimant made suicidal statements. In that case, the member was very calm and responsive, indicating that she was prepared to end the hearing and provide a written decision and to call appropriate medical personnel if needed. In the end, after assuring the claimant that the decision was going to be positive, calling for a break, and consulting with the claimant's counsel, the member was able to proceed with the end of the hearing.

Areas for improvement

• Issue agenda: ​The issue agenda was either wholly absent (n = 10) or inadequate (n = 2) in 37% (n = 7) of cases from the Eastern region, and in 14% (n = 5) of cases from the Central region.

  Inadequate or marginal issue agendas were characterized by a lack of specificity. For instance, in one case the member simply listed all of the elements of the refugee definition (alienage, well-founded fear, persecution, Convention grounds, IFA, state protection). In other cases, the members noted that, for instance, credibility was an issue without providing any additional guidance. Given that the purpose of the issue agenda is to ensure a focused and efficient hearing for both the claimant's counsel and the member, general statements such as those mentioned here are of little assistance.

• Member's questioning – relevance: ​Fifteen percent of cases scored a “1” on indicator #13, as the member in these cases failed to set out an issue agenda. The low scores are a result of indicators #11 and #13 being dependent on indicator #10 (Setting an Issue Agenda).

• Member questioning and narrowing of issues: ​Members narrowed the issues during the hearing in only 67% of the relevant cases. Although this score is low, it is somewhat misleading. In six cases, low scores were due to the failure of the members to properly identify issues at the outset. In two cases, the members appeared to expand rather than narrow the list of relevant issues without explicitly modifying the issue agenda. In several of the remaining ten cases, the issues set out initially were relatively narrow to start with and while narrowing them further may have increased the efficiency of the hearing, it likely had no substantive impact on the process.

  Of more importance is the failure of members to narrow the issues for final representations in eight cases. In most cases the member provided some minimal guidance while in several cases, members performed particularly well – clearly identifying which issues were no longer relevant or had been resolved and providing instructions to Counsel regarding which issues should form the focus of questioning and final representations.

Recommendation

• Members should be reminded of the requirement in Guideline 7 to establish a clear and detailed issue agenda in consultation with counsel and to explain expectations regarding how the hearing should unfold to both counsel and the claimant.
• Members should be reminded of the utility of clearly narrowing down the issues prior to questioning and final submissions by counsel.

2.4 Reasons state conclusions on all determinative issues

Why measure this:

The Supreme Court of Canada set the requirement for justifiability, intelligibility and transparency in a decision of an administrative tribunal. Through indicators #20 to #30 this study applies the Court’s requirement in the context of IRB decision-making.

Considerations:

These indicators are applied to all hearings.

General observations:

None identified.

Strengths

• Collectively, members met the requirements in terms of addressing all issues identified as determinative in the reasons, and providing conclusions that are based on the issues and evidence presented during the proceedings. In 52 cases, the reasons provided adequately addressed all of the issues that had been raised in the issue agenda or that were explicitly identified during questioning. In all cases evaluated, the conclusions were based on the claimant’s testimony, the documentary evidence submitted by the claimant, and the National Documentation Package. All of the cases from the Western region met the requirements of #20 and #21.

Areas for improvement

• Of the fourteen cases which were assessed as unsatisfactory for indicator #20, nine were entirely due to #20 being dependent upon the absence of an adequate issue agenda (as assessed under indicator #10). When issues are not initially identified, the subsequent scores are inevitably lower. Of these nine, six were in the Eastern region and three were in the Central region. Five additional cases from the Central region presented other problems: In two cases, the reasons provided did not address all of the issues identified as determinative at the hearing, while in three cases, new issues were raised in the reasons that had not been outlined in the issue agenda or clearly identified during the questioning. In two of those three cases, the failure to identify new issues prior to the reasons may have impacted the ability of the claimant’s counsel to respond appropriately during final submissions.

Recommendation

• None identified

2.5 Decisions provide findings and analysis necessary to justify conclusions

Why measure this:

The Supreme Court of Canada set the requirement for justifiability, intelligibility and transparency in a decision of an administrative tribunal. Through indicators #20 to #30 this study applies the Court’s requirement in the context of IRB decision-making.

Considerations:

Indicators # 22-24 are applied to all hearings, while #25-27 are applied on an as applicable basis.

General observations:

None Identified.

Strengths

• Detailed analysis: Members consistently met or exceeded expectations with respect to indicators # 22, #23, #24, #25. In the strongest examples, members explicitly set out the findings of fact and connected those findings to specific examples in the documentary evidence and the testimony provided during the hearing. Where issues arose regarding the credibility of documentary evidence or of testimony, members generally explained their concerns and the basis for their resulting determinations, outlining which evidence was deemed credible and which evidence fell short, as well as its impact on the decision.

  The majority of cases made excellent use of the NDP, often quoting from it or referring extensively to specific portions.

Areas for improvement

• Detailed analysis: In the cases that failed or only just met the requirements of indicators #22, #23, #24, and #25, low scores were due to a variety of factors including the need for greater use of clear examples (3 cases), failure to provide adequate explanations or reasons to support certain findings/conclusions (5 cases), or failing to make a finding with regard to an issue (2 cases). In one case, the member referred in the reasons to his own “specialized knowledge as to the different stories in the Bible and the central figures in the Bible.” Although the member used this knowledge to determine that the claimant was credible, the inclusion of this statement in the reasons and the tone of some of the questioning pertaining to religion raise some concerns regarding the potential perception of a lack of objectivity on the part of the member.
• References: As noted in the 2016-17 Quality Performance in the Refugee Protection Division Report,^{Footnote 2​} “[r]eferencing applicable legislation, caselaw or an IRB policy instrument, such as the Chairperson's Guidelines, informs the parties of the legal and policy authorities used by the member in reaching the decision.” The cases reviewed for this study varied greatly with regard to whether and to what extent the members satisfied this indicator. While the majority of cases did not raise any substantial concerns in this regard, referring only to s. 96 and 97 of IRPA, several observations are deserving of note.
• Of the 66 hearings reviewed, only eleven included any reference to jurisprudence. Of these, eight were from the Western region.
• Nine out of eleven relevant cases referred to the Chairperson's Guideline 9 on Proceedings Before the IRB Involving Sexual Orientation and Gender Identity and Expression in the reasons, while twelve out of thirteen referred to Guideline 4 – Women Refugee Claimants Fearing Gender-Related Persecution. In the majority of these cases, members mentioned that they were applying the guidelines without providing any additional information, for example indicating how the guidelines might impact their assessment of credibility. Providing specific explanation of the impact of the guidelines may help to increase the public's confidence in the Tribunal's competence to deal with these challenging cases and avoid the perception that members are merely mentioning the Guidelines without meaningfully considering and using them.
• Regional variations: Although many cases from all regions met the expectations with regard to references to jurisprudence, policy instruments and legislation, and the small sample size of the study makes generalized conclusions problematic, some regional variation was observed. In the cases assessed, the Western region stood out in particular with eight out of twelve cases exemplifying an exceptionally high standard of detail and thoroughness in this regard, referring to jurisprudence, as well as policy documents and clearly outlining the applicability of these sources.   

Recommendations

• A communique to members providing guidance on the reference to and use of rules, regulations, Chairperson's Guidelines, Jurisprudential Guides or persuasive decisions in their reasons could be of assistance and increase consistency across regions.
• Encourage members to provide additional explanation of their application of the Chairperson's Guidelines in their reasons when relevant.

2.6 Reasons are transparent and intelligible

Why measure this:

The Supreme Court of Canada set the requirement for justifiability, intelligibility and transparency in a decision of an administrative tribunal. Through indicators 20 to 30 this study applies the Court’s requirement in the context of IRB decision-making.

Considerations:

All indicators are applied to all hearings.

General observations:

None identified.

Strengths

• Consistent and Intelligible reasons: In virtually all cases across regions, members excelled in providing clear, intelligible, and organized reasons. Widely employed best practices include the use of headings and organizing the reasons chronologically or in accordance with the issue agenda. Members generally used plain language to provide concise reasons that addressed the issues of the case and were easily understood. A small number of cases (n=6) were noted for having reasons that were almost too concise and could have benefited from additional detail. However, this may be explained by noting that these cases were all decided in the claimant’s favor.

Areas for improvement

• Many of the transcripts of the reasons included minor spelling or grammatical errors or typos. In a few cases, errors were more substantial with words or portions of sentences missing or mis-transcribed. An additional edit or review of the transcripts would catch many of these issues and ensure a more professional, and accurate, result.

Recommendation

• Encourage members to use headings and to follow the issue agenda in organizing their reasons.
• Conduct a final edit or review of transcripts and reasons before they are released to the claimants or filed in the record.

3.0 Results for specific populations

3.1 Consideration for Sexual Orientation and Gender Identity and Expression

Why measure this:

To adequately satisfy Treasury Board Secretariat (TBS) guidance on Gender Based Analysis Plus (GBA Plus), a second checklist was developed for hearings involving persons of diverse sexual orientation, gender identity and expression (SOGIE) and is used in quality measurement reviews in all divisions. The checklist assesses the application of Chairperson’s Guideline 9: Proceedings before the IRB Involving Sexual Orientation and Gender Identity and Expression.

What was measured:

Target: 100% of hearings scored as compliant

Considerations:

SOGIE related hearings were proportionally represented in the sample, with ten hearings identified at the outset, based on the claim type of the file. Throughout the assessment, the reviewer identified one additional hearing as being SOGIE related. In total eleven SOGIE hearings were assessed using the checklist.  Not every hearing was assessed on each indicator. A response of “Yes” or “No” was provided only where the assessor found that the indicator was applicable. Where an indicator was only assessed in a handful of cases, conclusions and generalizations to the population are not made. Observations may still be provided for insight and reflection, but they are not representative findings.

General observations:

• Given the small number of cases reviewed, it is not possible to make generalizable assertions. Examples of good or bad practices are merely that – individual examples. Nevertheless, there is utility in drawing attention to certain points, not as a means of reprimanding the members involved but in order to increase awareness of potential pitfalls (including those that are inadvertent).

Strengths

• Guidelines: As noted previously, in nine of the eleven cases evaluated involving SOGIE considerations, the member explicitly mentioned the Chairperson's Guideline 9 in the reasons for the decision.
• Virtually all members appropriately omitted sensitive details such as the names of former or current partners (particularly those who remain in the country of origin) or details about sexual relations or incidents of sexual violence from the reasons.
• In nine of eleven evaluated cases, the member adequately referred to or explored in depth the relevant laws in force in the country of origin, particularly those criminalizing homosexuality, or the available information concerning state practice in the country of origin.
• In several cases, the member exhibited exceptional sensitivity to the claimant's position in the conduct of the hearing. In these cases, the member was demonstrably respectful, aware of potential cultural sensitivities around the issues being discussed, and willing to provide accommodations. Examples of accommodations provided include allowing breaks as needed and offering the claimant's partner the opportunity to testify first so that he/she could remain in the hearing room in order to provide emotional support.

Areas for improvement

• In only three cases did members mention the applicability of Chairperson's Guideline 9 during the hearing (prior to issuing the final reasons). Given the procedural implications of Guideline 9, this is a concerning oversight. Moreover, in the majority of cases in which the SOGIE Guideline was referred to in either the hearing or the reasons, there was little or no explanation of its implications for the proceedings.
• Drawing both on the general results and the results contained in the SOGIE checklist, in four of the evaluated cases, the member demonstrated a lack of sensitivity during the hearing. This lack of sensitivity was often reflected in the tone and manner of questioning. Although in none of these cases was the shortcoming egregious, given the sensitive nature of the issues discussed (including sexual violence, sexual relationships, sexual identity, medical history, family schisms, etc.) and the trauma and stigma that many claimants have been subject to, it is incumbent upon members to exercise a particularly high level of care in this regard in cases involving SOGIE issues. In the cases reviewed, several such points stand out:
• In one problematic case, the member drew a negative credibility inference regarding the claimant's sexual orientation. This finding was open to the member to make. However, in making this finding, the member appeared to rely on the assumption that if the claimant was homosexual, he would have had romantic same-sex relationships and that the casual same-sex relationships described were not indicative of the claimant being gay and were just “experimental”. As a consequence of this negative credibility finding, the member did not address the fact that homosexuality was criminalized in the claimant's country of origin or the potential impact of that criminalization on the claimant's narrative. 
• In another case, the member clearly struggled to find the appropriate words to address the claimant, referring to him at one point as a “flamboyant man” before correcting herself.
• Lastly, in yet another case the member questioned the claimant at length about whether he had gotten involved in the organized gay community in Canada. This line of questioning is directly cautioned against in section 6.1 of Guideline 9 and is not a recommended practice. In this case the subtext appeared to be a presumption that someone who is homosexual would become active in the gay community. Additionally, the member initially referred to the claimant as being bisexual despite there being no evidence to this effect in the narrative (the claimant was homosexual).

Recommendations

• Provide ongoing, compulsory training on the use of Guideline 9 and the impact that SOGIE considerations can have on the refugee determination process, including the ability of the claimant to testify.
• Encourage members to note and explain the applicability of Guideline 9 at the outset of SOGIE related hearings.
• Provide guidance to members on how to explicitly implement the principles contained in Guideline 9 in their hearing, as well as in their reasons. 

3.2 Women fearing gender-based violence (WFGBV)

Why measure this:

To adequately conduct Gender Based Analysis Plus (GBA Plus​), the evaluation unit identified hearings related to gender-based violence and gender-related persecution within the sample, to assess the quality of decision making, as well as the application of relevant decision-making instruments such as Chairperson’s Guideline 4: Women Refugee Claimants Fearing Gender-Related Persecution. Guideline 4 came into effect in September 1996, to encourage and promote consistency in dealing with refugee claims made by women fearing gender-related persecution.

The Evaluation Unit compared the distribution of scores in each result section (2.1-2.6) for hearings involving women fearing gender-based violence (WFGBV) to those of the entire sample.

Considerations:

Hearings of women fearing gender-based violence (WFGBV) were proportionally represented in the sample, based on the claim type of the file, with eleven hearings identified at the outset. Claim types that were used to identify cases included, among others: domestic violence, female genital mutilation, forced marriage, forced kidnapping/prostitution, and sexual violence. Throughout the assessment, the reviewer identified two additional hearings that involved WFGBV or where Guideline 4 was considered or applied. In total, thirteen hearings were identified and assessed. The assessor was asked to provide observations on those hearings.

General observations:

• None Identified

Strengths

• In twelve of the thirteen relevant cases evaluated, members made explicit reference to Chairperson’s Guideline 4 – Women Refugee Claimants Fearing Gender-Related Persecution, in their reasons.
• In all but two of the relevant cases, members met or exceeded expectations with regard to treating all parties involved with sensitivity and respect. In nine out of thirteen cases, the member scored a “3”. Members were clearly alert to the potential difficulties of testifying before the tribunal for women seeking protection from gender-based violence and frequently explicitly acknowledged this vulnerability. In many cases, members also made it clear to claimants that they could request a break at any time they felt it was necessary or suggested taking a break on their own initiative when claimants became overwhelmed by emotion. In the one case where the claimant was identified as a vulnerable person in accordance with Chairperson’s Guideline 8, the member allowed questioning to be done in reverse order; so that the majority of the questioning was done by counsel with whom the claimant was comfortable. The member also suggested frequent breaks, and explicitly acknowledged to the claimant that he/she understood the difficulty of testifying given the nature of the claims (as involving domestic violence).

Areas for improvement

• While the majority of members made direct reference to Guideline 4 in their reasons, in only two cases out of the thirteen evaluated did members refer to that Guideline during the hearing. Given the procedural implications for the hearing of Guideline 4, indicating its application early in the hearing may be useful as a reminder both for members and for the parties of its importance.
• Although members generally exhibited sensitivity and respect when dealing with claims involving women fearing gender-based violence, in none of the cases did they provide any explanation of how Guideline 4 was applied or what, if any, impact it had on their decision-making process. 
• As noted in 2.1 ‘Timely and Complete Pre-Procedure Readiness’, in approximately half of cases reviewed, the Hearing Disposition Form (HDR) was incomplete and failed to note that the case involved Guideline 4, even when the Guideline was directly referred to in the reasons. 

Recommendation

• Provide ongoing, compulsory training on the application of Guideline 4 – Women Refugee Claimants Fearing Gender-Related Persecution.
• Encourage members to make explicit and detailed reference to Guideline 4 when applicable during the hearing as well as in their reasons.

Appendix A - Checklist

Appendix B - SOGIE quality review checklist: Performance indicators and rating guide