Quality Performance in the Refugee Appeal Division 2020 to 2021

​​​​On this page

• 1.0 Context
• 2.0 Performance results
  
• 2.1 Reasons are complete
• 2.2 Reasons are transparent and intelligible
• 2.3 Supplementary questions
• 3.0 Results for specific populations
• 3.1 SOGIE cases
• Annex A – Checklist
• Annex B - SOGIE quality review checklist

1.0 Context

This report describes the results of the measurement of quality in decision-making in the Refugee Appeal Division (RAD).

Sample methodology

The study reviewed 70 out of a possible 1282 paper appeals that were finalized between October 1 and December 31, 2020 (the assessment period). The study assessed appeals decided on their merits after a review of the information on file (documents provided by the appellant, Minister, and the RPD record). There were no RAD hearings conducted in respect of the assessed appeals. The appeals were randomly selected in proportion to region, language of proceeding, ministerial intervention status, outcome, and specially selected case types, including cases related to Sexual Orientation and Gender Identity and Expression (SOGIE)^{Footnote 1}, and women fearing gender-based violence (GBV). Members who had been deciding cases for less than 12 months from the start of the assessment period were excluded from the sample. Within the sample, six cases related to Sexual Orientation, Gender Identity and Expression (SOGIE) were further analysed through a particular SOGIE lens. However, given the small sample size observations are not generalizable.

The following charts illustrate the sampling makeup, which is proportionately representative of the population data:

Assessment Methodology

This qualitative assessment was performed by an independent reviewer who is the Research Chair in Administrative Law and Governance at the University of Ottawa, a leading commentator on Canadian administrative law, and an expert in decision-making in the administrative state. The reviewer examined all necessary evidentiary and administrative materials on file and assessed these against qualitative indicators in a checklist developed by the Strategic Planning, Accountability and Reporting (SPAR) Directorate and approved by the Deputy Chairperson of the RAD (see Annex A). The checklist assesses twenty-two indicators across three categories:

1. Reasons are complete
2. Reasons are transparent and intelligible
3. Supplementary questions

Eighteen of the indicators were mandatory for assessment, and four were assessed only when applicable. Each performance indicator is assessed along a 1-3 rating scale and, for supplementary questions related to SOGIE cases, a categorical yes-no scale was used.

The 1-3 rating scale is as follows:

1=Does not meet expectations: The quality requirement was not met. The evidence showed one or more key instances where the proceeding or reasons would have markedly benefited had this requirement been met. There may have been an effort to apply the requirement, but the level of achievement fell short of expectations.

2=Meets expectations: This is a level of acceptable achievement. On balance, the member satisfied this quality requirement though there is margin for minor improvement.

3=Exceeds expectations: This is a level of consistent, above-average achievement. The evidence shows a grasp of the quality requirement and an understanding of its importance to a high-quality proceeding or decision, as the case may be.

Results are also expressed as a percentage of cases that meet expectations, by obtaining a score of 2.0 of higher.

In addition, as part of Gender Based Analysis Plus (GBA+), a second checklist was developed for cases involving SOGIE persons of diverse sexual orientation, gender identity and expression (SOGIE) and is used in quality measurement reviews in all four of IRB’s divisions when applicable (see Appendix B).

Considerations/limitations

Results are accurate to within ±10% percent, 18 times out of 20. This margin increases when data is broken down by region or claim type. However, the goal of the study was not to generate statistics but to identify areas of strength, concern, and patterns in decision-making quality.

This study acknowledges the inherent limitations of qualitative research, which does not generate precise data as does quantitative research. To mitigate the inherent limitations of qualitative research, detailed performance indicators were provided to the reviewer to help focus the assessment. To ensure quality and consistency in the assessment, a reviewer was selected based on their in-depth knowledge of the IRPA, refugee and immigration matters, and administrative law. Moreover, a small sample size limits the inferences that may be made about the broader caseload. Where sample sizes are too small, observations or recommendations may still have been provided but these are not based on representative findings.

This report aims to provide a perspective to improve the Division’s performance overall. The evaluation unit provided the statistics found in the table accompanying each result section as well as the information in 1.0 “Context”. However, the findings in this report, including all strengths, areas for improvement and recommendations are solely those of the reviewer. Their observations do not lend themselves to firm conclusions on legal matters such as the correct application of the law, the weighing of the evidence, or the fairness of the proceedings from a natural justice perspective. Only a court reviewing the case can arrive at such conclusions.

2.0 ​Performance Results

What was measured

Each performance result in sections 2.1-2.3, contains a table representing the number of cases assessed for each indicator, and the percentage of assessed cases that met expectations by scoring a 2.0 or higher for that indicator. The number of cases assessed are provided for reference and context only.

The primary performance target for this assessment is for 75% of cases to meet quality standards. The quality standard is that the case achieves a score of 2.0 or higher in at least 80% of all standard indicators assessed within that case. This target was achieved with 90% (63 out of 70 cases), meeting the expectations. In other words, over 90% of appeals reviewed either met or exceeded the quality standard.

Where an indicator had many cases that did not meet the target, it is addressed in the reviewer’s observations following the table (Areas for Improvement).

2.1 Reasons are Complete

Why measure this

The Supreme Court of Canada has affirmed that justifiability, intelligibility, and transparency are important hallmarks of a reasonable administrative tribunal decision. Indicators 1 to 17 of this study applies these principles to the context of IRB decision-making.

Considerations

Indicators 3, 4, and 11 are only considered if applicable. Remaining indicators are considered in all paper appeals.

General Observations

• In general, decisions were well-written and satisfied the criteria of justification, transparency and intelligibility.
• Members have clearly been well-trained in the art of point-first and issue-driven writing, which requires the writer to clearly state the points to be decided, with their analysis then flowing through the legal framework and their findings of fact to a clear, unambiguous conclusion. Issue-driven writing involves the production of decisions which are thematic, rather than chronological. Taken together, point-first and issue-driven writing in an administrative law context leads to decisions with the following form: (1) identify the relevant issues; (2) provide contextual factual information about the issue; (3) outline the applicable legal framework and standards; (4) discuss the evidence and arguments presented; and (5) set out the conclusions having applied the legal framework and standards to the facts. Engaging in point-first and issue-driven writing makes administrative decisions much easier to read.
• In addition, Members appreciate the importance of writing for the losing party. Writing for the losing party requires decision-makers to ensure that they have touched on all relevant issues raised by the unsuccessful party and clearly communicated the reasons for the conclusion. Relatedly, Members appreciate the importance of plain language writing, especially given the context of the work of the RAD, where the appellant is often a non-native speaker of English or French. Plain language writing means writing in short sentences, using simple words and avoiding the passive voice.
• It is sometimes challenging for Members to apply the appropriate standard to findings of the RPD, as on the one hand they are to make an independent assessment of the claim but on the other hand this assessment is framed by the reasons of the RPD and submissions of counsel.

Strengths

• Indicator 1: Members understand and accurately articulate the sui generis standard of review applied by the RAD. This standard does not assess the reasonableness of the RPD's decision, but whether its decision was correct.”
• Indicators 3 and 4: Members had no difficulty articulating and applying the test for the admission of new evidence and the holding of an oral hearing. Reasons on new evidence/oral case were of a very high standard, as they clearly and concisely explained the legal framework and Members' findings.
• Indicator 5: Members almost invariably addressed the issues they were asked to address.
• Indicator 6: Members generally identified determinative issues and focused on them. Difficulties arise where a claimant has a residual profile which was not addressed by the RPD, but in general Members were alive to these difficulties In one example, the RPD decision-maker had not thoroughly canvassed the issues raised by the refugee claimant.  In determining the appeal, the Member did an excellent job of focusing on forward-looking risk as the determinative issue and treating the appellant's residual profile as a potential victim of gender-based violence by reference to Chairperson Guideline 4: Women Refugee Claimants Fearing Gender-Related Persecution.  The Member's thorough independent assessment of the merits of the claim ensured finality.
• Indicators 7 and 8: Members plainly understand the need to make clear and unambiguous findings of fact and made efforts to do so even in respect of appeals where the underlying facts were somewhat unclear or ambiguous. For example, the fundamental difficulty with one claim was that insufficient evidence had been provided by the appellant. The Member appropriately considered the documentary and testimonial evidence and made clear and unambiguous findings of fact that the appellant had not made out her case that her persecutors would locate her in one of the identified Internal Flight Alternatives.
• Indicator 9: Members effectively ‘write for the losing side', explaining why a claim was not successful.
• Indicator 10: Members are excellent at identifying the sources of law relevant to making determinations.
• Indicator 11: Members also demonstrated cultural competence in most appeals.

Areas for improvement

• Indicator 2: Members should attempt to summarize their reasons for allowing or dismissing an appeal in a sentence or two in their opening paragraphs. On balance, the summaries were very good, but it would be helpful for the reader (including the successful or unsuccessful appellant) if the key points were highlighted at the outset of the Member's reasons.
• Indicator 8: Members should take care to link clear, unambiguous findings of fact to evidence. In particular, Members should avoid over-reliance on RPD findings: for example, in one case, the Member relied on RPD findings without explaining clearly why he found them convincing; the resultant difficulty was that the underlying evidence supporting the factual findings was not clearly identified. In another case, the RPD's findings on credibility were simply adopted in conclusory fashion. Linking findings of fact to evidence is especially important in cases where the underlying claim is weak, such as in cases where the reasons for the weakness of the claim may be difficult to articulate. In such situations, reliance on RPD findings will be more problematic, as it may suggest that the RAD did not engage in an independent assessment of the claim. Ultimately, Members need to be responsive to the reasons of the RPD and, in reaching independent conclusions, and base those conclusions on the evidence.
• Indicator 10: Some Members failed to refer, or give appropriate weight to, the presumption of truthfulness and the presumption of regularity of documents. When a refugee claimant swears to the truth of allegations, these are presumed to be truthful; and documents issued by foreign governments are presumed to be authentic. For example, in one case involving a claim of a forward-facing risk of persecution, due to the risk of being sterilized in the future for having violated China's one-child policy in the past, the presumption of regularity of documents was cited by the appellant but the Member did not explain why or how the concerns identified cumulatively overcame the presumption.
• Indicator 12: Sometimes the way an appeal is framed can make it difficult for the Member to adequately consider all relevant issues, as the appellant may simply take issue with the RPD's analysis without providing a clear argument, supported by the evidence in the record, as to how the RPD may have erred. A particularly striking example is a case where the appellant asserted that the RPD's findings on each prong of the IFA test were wrong but did not point to any evidence in this regard. Conducting an independent assessment of a claim whilst also being responsive to the RPD's findings and the submissions of the appellant appeared to be a challenge in some such cases.
• Indicators 12 and 16 (The reasons are as short and economical as possible taking into account the complexities of the appeal and volume of evidence): A related difficulty is that submissions will sometimes be made on points which need not be addressed because another point is determinative of the appeal. For example, in one case, the credibility analysis would have been much shorter and just as definitive but for the Member's responsiveness to the appellant's points that were not determinative. In one case, the analysis of credibility was long and detailed (in response to the appellants' points) but boiled down to the fairly simple proposition that one of the appellants could not credibly establish that she had worked for an NGO. And in two other cases, credibility was determinative, but the Member nonetheless addressed other arguments made by the appellants/claimants in those files. Whilst this responsiveness is laudable, it also makes the Member's reasons longer and less economical. Given the resulting effect on length and economy, Members should explain why they have addressed points which are not necessary to the outcome.

Recommendation

• Continue training on high-quality decision writing, with an emphasis on point-first and issue-based writing, writing for the losing side and providing a pithy summary of the decision in the opening paragraphs.
• Provide training on writing convincing reasons in cases where the underlying claim is weak or poorly articulated.
• Provide training on presumptions (such as the presumption of truthfulness) which are central to Members' work, to ensure that Members appreciate the content of the presumptions and the factors relevant to rebutting the presumptions in the context of a particular case.
• Highlight, in continuing professional development seminars, decisions where the Member has done a particularly good job of linking factual findings to evidence.
• Consider developing strategies, perhaps in the form of internal checklists, to ensure that all relevant issues are considered on appeal.
• When Members address points in the interests of responsiveness, they should explain that the points are not necessary to the outcome.

2.2 Reasons are Transparent and Intelligible

Why measure this

The Supreme Court of Canada has affirmed that justifiability, intelligibility, and transparency are important hallmarks of a reasonable administrative tribunal decision. Indicators 1 to 17 of this study applies these principles to the context of IRB decision-making.

Considerations

All indicators are applied to all paper appeals.

General observations

• Members write in plain language.
• Members’ decisions are generally clear, concise and easy to understand.
• Reasons are almost invariably of an appropriate length.
• Members typically write in such a way as to explain the decision to the losing side.
• There is a tension between writing reasons which are short and economical but also responsive to the issues raised (which often go beyond those which are determinative of the appeal).
  

Strengths

• Indicators 13-17: Members write point-first, issue-driven decisions, with a focus on writing for the losing side. Their strengths in this area contribute to their excellent performance against these criteria.  In the cases reviewed, there were decisions in which members demonstrated that they had written for the losing side by making clear reference to the applicable legal framework and standards. In one example, the Member provided issue-based, point-first writing, with clear findings of fact, a clear summary and a succinct explanation of why internal flight was the determinative issue in the appeal.
• Indicator 16: Members are generally good at identifying determinative issues and writing only on those issues, in order to produce short reasons.

Areas for improvement

• Indicator 14 (concision): Members appeared to have challenges in addressing appeals where the appellant made arguments which were not, strictly speaking, relevant but the Member nonetheless felt an obligation to be responsive to the appellant's arguments. For example in one case where credibility was determinative other points were considered in a responsive fashion.  In another case credibility was comprehensively analyzed in response to the appellant's arguments, though credibility could have been addressed more concisely.
• Indicators 14 (clarity), 15: Members had more difficulty in providing clear and concise reasons in cases where the claim for refugee status was relatively weak or poorly articulated, or the RPD decision was weak or poorly articulated. In such circumstances, reasons are apt to lack clarity and a logical line of reasoning. As recommended above in section 2.1 of this report, the key in such circumstances is to take care to make clear and unambiguous findings of fact. Once this is done, the Member will be better able to provide clear, logical reasons with greater ease. For example, in one case, the analysis of credibility consisted of a discussion of the RPD's erroneous negative inferences. The reader surmises that the presumption of truthfulness applied and therefore the appellant was credible, but the Member did not clearly relate the grounds for the finding of credibility to the evidence the appellant provided to support his identity as a bisexual male. The reasons would have been clearer and more logical had the Member expressly referenced the presumption of truthfulness and/or relate the Member's legal conclusions to findings of fact based on the appellant's evidence.
• Indicator 15: Members occasionally struggled with overlaps between issues such as credibility, internal flight alternative and state protection. In one case, for example, there was significant overlap between the Internal Flight Alternative and State Protection analyses. In such instances, careful cross-references are appropriate.
• Indicator 16: Members were generally good at identifying determinative issues, thereby reducing the length of their reasons. There are, nonetheless, examples of appeals which could have been disposed of with shorter reasons, especially where Internal Flight Alternative was determinative or where the Members wrote both on credibility and Internal Flight Alternative; or where a focus on Internal Flight Alternative would have reduced the length of the reasons.
• Indicator 17: Writing for the losing side can sometimes be challenging, but where an issue is raised and is relevant, it will be appropriate to deal with it. Where an issue is raised and is not relevant, the Member does not always explain why the issue is not addressed. When an issue raised by the appellant is not addressed, the Member should explain why the issue was not relevant and thus not discussed.

Recommendations

• Strategies should be developed for writing transparent and intelligible reasons in cases where the underlying claim is somewhat weak or poorly articulated.
• Where overlap arises between different issues, Members should take care to cross-reference paragraphs of their reasons which address the same or similar issues and to explain why the treatment of the issues is (or is not) distinct.
• Members should also consider how to strike an appropriate balance between shortness and economy, and responsiveness. There is no easy answer here, as there is an inevitable tension between shortness and economy, and responsiveness. One possibility is to briefly explain why certain arguments were not responded to, because of their lack of relevance.
• When an issue is raised by an appellant but is not relevant, Members should provide an explanation in the reasons as to why the issue does not need to be addressed.

2.3 Supplementary Questions

Why measure this

The supplementary questions’ section allows the RAD to incorporate new indicators to the annual RAD QMI review that may not fit into other sections, and contributes to a responsive, relevant, and robust assessment of quality.

Considerations

Indicators 21 is only considered if applicable. Remaining indicators are considered in all paper appeals.

General observations

• Members were aware of the need to conduct an independent assessment of the claim, but there is a tension between the need to be responsive to the RPD's findings and the submissions of the appellant and the need to conduct an independent assessment. On the one hand, the RAD is designed to conduct an independent assessment of the underlying claim and thereby bring finality to the question of the appellant's refugee status. This requires consideration of all aspects which are relevant to the determination of refugee status. On the other hand, however, appeals before the RAD are framed by the appellant, who must identify errors in the RPD's analysis as a condition precedent to succeeding on appeal. This leads some appellants to focus on errors in the RPD's analysis rather than the question of their entitlement to refugee status, creating two difficulties. First, the appellants may make arguments which are not relevant to the determination of refugee status but to which nonetheless the RAD may feel obligated to be responsive: this will make for longer, more complex reasons (a point also discussed above in section 2.2). Second, the appellants may fail to make a positive case for recognition of refugee status: this makes it more difficult for the RAD to conduct an independent assessment of the claim and bring finality to the claim.
• Members disposed of appeals where they could and thereby achieved finality in most cases. For example, in one case, the treatment of forward-facing risk as the determinative issue, combined with the analysis of residual risk, ensured finality.

Strengths

• Indicator 21: In terms of intersectionality and cultural competency, Members wrote excellent reasons which were sensitive to the particular context of the appellant's circumstances. A good example is a case where the Member was sensitive to cultural context and the trauma the appellant suffered from the alleged events.

Areas for improvement

• Indicator 18: Some appeals provide platforms for providing guidance to the RPD, especially where an RPD member seems to have misunderstood basic procedural or substantive legal requirements. However, Members missed opportunities to write reasons which would clarify these requirements for RPD members who evidently need clarification.
• Indicator 20: There is a tension between the need for an independent assessment, which brings finality to a refugee claim, and the need to be responsive to the RPD's findings and the appellant's submissions.  In some cases, responsiveness led Members not to perform an independent assessment, for instance by simply responding to the RPD's findings and the appellant's submissions and failing to notice that a claim should be decided on grounds not raised by the RPD's findings or the appellant's submissions. Members should ensure that they conduct an independent assessment even where the findings and/or submissions are not helpful in doing so.

Recommendations

• The RAD should reiterate to Members the need to conduct an independent assessment rather than simply a review of alleged errors by the RPD: alleged errors open the door to an appeal, but an independent assessment should be conducted to ensure finality. The RAD (and perhaps the IRB as a whole) should consider reformulating its advice to appellants and maybe even the forms provided to appellants with a view to signaling that the RAD's role is not simply to correct errors made by the RPD but to conduct an independent assessment of the appellant's entitlement to refugee status.
• Members should be reminded that their decisions can give useful guidance to the RPD, for example in assessing the genuineness of religious convictions or the authenticity of a particular type of document.
• Where the RPD's findings and/or the appellant's submissions do not assist the Member in performing an independent assessment of the refugee claim, the Member should explain why the findings and/or submissions are unhelpful and proceed to perform an independent assessment of the claim.
• Training on intersectionality and cultural competence are evidently achieving excellent results and, accordingly, should continue.
• Strategies should be developed to address the tension between the need for an independent assessment and the need for responsiveness.

3.0 Results for specific populations

3.1 Consideration for sexual orientation and gender identity and expression

Why measure this

For the purposes of Gender Based Analysis Plus (GBA+), which is a priority for the IRB as well as within the Government of Canada as a whole, a second checklist was developed for cases involving persons of diverse sexual orientation, gender identity and expression (SOGIE). The checklist assesses the application of Chairperson's Guideline 9: Proceedings before the IRB Involving Sexual Orientation and Gender Identity and Expression.

What was measured

Only six SOGIE cases were identified within the sample, however, not all indicators were applicable to every case. N/A ratings were not included in final calculations for ‘# of cases assessed’ or in ‘% of cases compliant.’ Due to the small number of SOGIE cases and indicators assessed, findings cannot be generalized to other SOGIE cases and are included as observations only.

Target: 100% of cases scored as compliant

General observations

• There were few SOGIE cases in the sample. Accordingly, caution must be exercised in drawing general lessons from this review. With that caveat in mind, the treatment of SOGIE issues was generally excellent.

Strengths

• For the most part, the treatment of SOGIE cases was excellent. Members are clearly alert, alive, and sensitive to the need to give appropriate consideration to the unique characteristics of appellants who fall under the SOGIE rubric. For example, in one case the Member took a generous approach to what the RPD considered to be inconsistencies or omissions in the evidence, an approach justifiable due to the appellant's SOGIE characteristics.
• Members separated files appropriately, addressed appellants by their desired mode of address where appropriate, universally refrained from relying on stereotypes and unanimously demonstrate awareness of the possibility that laws of general application will be used to target individuals with diverse SOGIE.

Areas for improvement

• Members could have considered measures to maintain anonymity of individuals with whom SOGIE appellants interacted. For example, in one case it would perhaps have been appropriate for the member to refer to the appellant's alleged male partner as T rather than by his full first name, given that anyone who knew the appellant's name would likely be able to track down T based on the address they shared. In our interconnected age, even minor details might facilitate persecution.
• Members were not always alive to how SOGIE characteristics could result in the giving of vague or contradictory testimony, or the perception that testimony was vague or contradictory. For example, in one case the Member could have taken the opportunity to consider whether a more generous, SOGIE-informed approach to considering delay and letters of support would be appropriate.

Recommendations

• The RAD should consider putting in place protocols for ensuring that identifying information is removed from decisions under the SOGIE rubric. A large and liberal approach would be appropriate in this regard, given the possibility that information may be used by persecutors.
• The RAD should reinforce Members' awareness of SOGIE characteristics, especially the risk that traumatic experiences will lead to the giving of testimony which may be perceived to be vague or contradictory.

Annex A – RAD performance indicators

Annex B – SOGIE quality review checklist: Performance indicators and rating guide