Evaluation of Member Training Immigration and Refugee Board of Canada

Table of contents​

1. List of Abbreviations Used in Report
2. Executive Summary
   1. Overview of the Immigration and Refugee Board of Canada
   2. IRB Organizational Structure
   3. Evaluation Methodology
   4. Findings
   5. Recommendations
3. I Introduction
   1. 3.1 Overview of the Immigration and Refugee Board of Canada
   2. 3.2 IRB Organizational Structure
   3. 3.3 Evaluation Methodology
4. II Findings
   1. Overview of Member Training and Professional Development at the IRB
   2. Roles and Responsibilities and Accountability
   3. Approaches to Planning and Accountability
   4. Effectiveness of Training and Professional Development Activities
   5. Adequacy of Resources
   6. Alignment with IRB Mandate and Priorities
   7. Best Practices and Alternatives
5. III Conclusions and Recommendations
   1. Conclusions
   2. Recommendations
6. Appendix A – Research Matrix
   
7. Appendix B – Interview Guide
8. Appendix C – Survey Questionnaire
9. Appendix D – Comparative Anaylsis Interview Guide

List of Abbreviations Used in Report

• ADC – Assistant Deputy Chair
• CBSA – Canada Border Services Agency
• DC – Deputy Chair
  
• ERGA – Experience-Reflection-Generalization-Application
• FTE – Full-time equivalent
• IAD – Immigration Appeal Division
• ID – Immigration Division
• IRB – Immigration and Refugee Board of Canada
• IRPA – Immigration and Refugee Protection Act
• LPD – Learning and Professional Development
• LPDD – Learning and Professional Development Directorate
• NJI – National Judicial Institute
• RAD – Refugee Appeal Division
• RPD – Refugee Protection Division
• SST – Social Security Tribunal

Executive Summary

This report presents the findings of an Evaluation of Member Training^{Footnote 1} at the Immigration and Refugee Board of Canada (IRB). The focus of the evaluation was on “adjudicative” training provided to IRB members, which consists of training sessions and related professional development activities that aim to improve the effectiveness and/or efficiency of members in fulfilling their role as decision makers. The evaluation thus excluded the examination of other types of “corporate” training provided to members (and to IRB staff), such as training on harassment prevention.

The purpose of the evaluation was to review the design, development, and delivery of adjudicative training services provided to members in terms of their effectiveness, efficiency, best practices, and lessons learned.

The evaluation was conducted by Kelly Sears Consulting Group on behalf of the IRB’s Planning, Evaluation and Performance Measurement Directorate between November 2017 and April 2018.

Overview of the Immigration and Refugee Board of Canada

The mission of the IRB, on behalf of Canadians, is to resolve immigration and refugee cases efficiently, fairly and in accordance with the law.^{Footnote 2}

The IRB carries out its work in three regional offices located in Toronto (Central Region), Montréal (Eastern Region) and Vancouver (Western Region). The Central Region is responsible for Ontario, excluding Ottawa; the Eastern Region for Quebec, Ottawa and the Atlantic provinces; and the Western Region for the western provinces, the Northern services and the Calgary office.

The IRB delivers on its mission through four divisions: Refugee Protection Division (RPD), Refugee Appeal Division (RAD), Immigration Division (ID), and Immigration Appeal Division (IAD). All four divisions hold hearings in the regions and are assisted by registry services and corporate support. Hearings are also held in a small number of itinerant locations. The IRB's National Headquarters are located in Ottawa.

Through the work of each division, the IRB strives to deliver a simpler, more accessible and expeditious form of justice than that provided by the courts. The IRB applies the principles of administrative law, including those of natural justice, in its proceedings. Its decisions are rendered in accordance with the law, including the Canadian Charter of Rights and Freedoms. The IRB is committed to fairness in all aspects of its work. Each case is decided on its own merits by independent decision-makers.

During 2016-17 the IRB delivered on its mandate with a budget of approximately $115M and 976 full-time equivalents (FTEs), including members. Table 1 below summarizes the number of members by division and region, excluding the Deputy Chairs and Assistant Deputy Chairs. The RPD has by far the largest number of members in all regions. Likewise, the majority of members are located in the Central Region.

IRB Organizational Structure

Members of the IRB are either appointed through the Governor-in-Council (GIC) process, as is the case for RAD and IAD members, or hired under the Public Service Employment Act, as is the case for RPD and ID members and all other IRB employees. All members report through their respective divisional ADCs and DC to the Chair. Legal Services report directly to the Chair. The various core internal services (human resources, labour relations, finance, accommodations, communications, etc.) that support the four divisions report to the Executive Director.

Evaluation Methodology

The evaluation covered the adjudicative training provided to new and existing members over the period from March 31, 2013 to December 2017.

During the planning phase of the evaluation, Kelly Sears worked with the IRB Planning, Evaluation and Performance Measurement Directorate to finalize the methodology and refine the data collection instruments. The evaluation methods included:

Key Informant Interviews: A total of 41 in-person and telephone interviews were undertaken with IRB representatives, covering Headquarters and all of the divisions and regions. The interview guide is provided in Appendix B.

Online Survey of Members: A total of 113 survey responses were received, for a response rate of 47%, which is higher than the typical response rates for internal surveys reported by online survey companies. With regard to length of service of the members who responded, 45% became members in their current divisions in 2016 or 2017, 34% between 2011 and 2015, and 21% prior to 2011. RPD, RAD and IAD distributions were similar to the overall pattern, whereas a large majority of ID respondents (77%) were appointed to their current positions prior to 2011. The survey questionnaire is provided in Appendix C.

Comparative Analysis: Examined alternative approaches and best practices relating to training and professional development in other similar organizations. Two organizations were selected for the comparative analysis – the National Judicial Institute (NJI) and the Social Security Tribunal (SST). The comparative analysis involved the conduct of an in-person interview with a senior manager knowledgeable about adjudicative training in each organization, along with a review of documentation and literature available online. The interview guide for the interviews with representatives of the two comparative organizations is included in Appendix D.

Review of Documentation: The documents reviewed included annual departmental plans and results reports, sample training materials, and previous studies (particularly an organizational review of the IRB Learning and Professional Development Directorate in 2005).

Findings

Roles and Responsibilities

The identification of member training needs and the development and delivery of training services is mainly the responsibility of two groups: the four divisions (which define training needs and may be involved in the design and delivery of training sessions) and Legal Services (substantive law experts who also help to identify member needs and develop and deliver the training). The DCs and ADCs are viewed by Legal Services as the primary clients for member training. Aside from helping to re-design new member training within IAD, the LPDD has had limited involvement in member training in recent years. LPDD indicated that it no longer has the resources to support this activity. Most key informants indicated that the roles and responsibilities for member training are not clear; for example, if LPDD is no longer to be involved in the design and delivery of member training, then this should be communicated.

Identification of Member Training Needs

The identification of member training needs is a collaborative process involving the four divisions and Legal Services, and, as indicated above, the divisions take the lead. The IRB does not have a national, nor divisional, professional development committees that would identify the common training needs across divisions and coordinate the provision of training across the organization. In general terms, DCs appear to be satisfied with the current informal approaches to the identification of training needs.

Overall, two-thirds of members surveyed are satisfied with the needs identification process, indicating that there is room for improvement. Suggestions mostly related to systematically soliciting member input and establishing individual training plans as part of the member performance review process.

Planning, Design and Delivery of Member Training

In general, the ADCs or DC of each division have the final say on the agenda for each training session, including new member training as well as ongoing professional development. There is agreement that the design of training sessions in all regions is done largely by Legal Services with assistance/input from members. Little training is planned or designed in collaboration with LPDD. Three divisions – RPD, IAD and RAD – have redesigned their new member training programs based on adult learning principles. As of the time of our research in early 2018 the ID had not decided whether to adopt the new approach. Ongoing training is delivered almost exclusively by Legal Services in all divisions and regions.

Monitoring and Evaluation

There is no formal process to monitor and evaluate the extent to which training is applied and desired performance outcomes achieved.

Effectiveness of Training and Professional Development

DCs are generally satisfied with the provision of new member training, and the approaches introduced in the RPD, RAD and IAD are perceived to be better equipping new members to understand and interpret the applicable substantive law. They are less satisfied with the provision of ongoing professional development, in terms of equipping members to manage hearings, make decisions and manage caseloads, and the coordination of planning and information sharing between regions. Most DCs also identified that bringing members together for national events can be very productive, in that a lot of learning is a function of both subject matter and related discussions between members.

The new re-designed new member training programs in the three divisions based on adult learning principles are seen as an improvement to the previous approaches. Some interviewees were concerned with the increased time and resources required to train new members using the new approaches. An analysis of trends in new member productivity in the RPD Central Region indicates the new approach may be helping new members achieve full productivity in a shorter length of time, although it is too early to draw a definitive conclusion. The findings of the survey of members show that a vast majority of members were satisfied with the training they received, particularly for new member training and training covering the substantive law and knowledge required to inform decision-making. Ongoing training related to conducting hearings and making decisions and managing caseloads and workflows is not entirely meeting the needs of members. Similarly, significant proportions of members indicated that they were receiving insufficient amounts of these types of training.

The IRB conducts most training in-person and this is the mode most preferred by members. The IRB is increasingly experimenting with videoconferencing and webinars, with varying degrees of success. A formalized process of mentoring is also being used increasingly as part of new member training and this is viewed favorably by members.

Adequacy of Resources and Operational Efficiency

The evidence suggests that the level of resources devoted to member training and professional development is adequate. However, existing resources could be used more effectively and efficiently through better coordination and collaboration across divisions and, within divisions, across regions. Legal Services has a need for increased administrative support for training. Increased funding for translation is required.

There is a lack of coordination within divisions and nationally. There is no central repository of training materials which limits the sharing of materials across divisions and regions. Addressing these issues would result in greater effectiveness with a reduced level of resources devoted to training overall.

Recommendations

Our evaluation of the IRB’s learning and professional development function found that, while majorities of the participating members were satisfied with the training and professional development they received, certain needs are not being well-met and opportunities exist to improve and better integrate training and professional development approaches. The most important of these needs are to:

• Ensure that training and professional development is built around members teaching members, supported by suitable subject matter and delivery expertise.
• Implement a systematic and integrated approach to new member training and ongoing professional development.
• Increase the provision of training on:
• Conducting hearings and decision writing.
• Managing caseloads and workflows.
• Incorporate adult learning approaches into the design and delivery of new member training and ongoing professional development.

However, in order to achieve an optimal combination of effectiveness and efficiency, responses to these needs should positioned within a broader strategic context for managing, planning, designing and delivering member adjudicative training. Our recommendations are thus grouped under the major parameters for this strategic context.

1. Strengthen the oversight and direction for member training

• Each division should have a national committee to provide direction for, and oversight of, both new member training and ongoing professional development for their members, with a mandate to:

• Identify and prioritize members’ training needs.
• Develop annual (or biennial) training and professional development plans, and accompanying budgets, for review and approval by the senior leadership of the IRB.
• Work collaboratively with the other divisions’ training and professional development committees to identify common training needs among members that can be addressed using IRB-wide approaches and courses, as part of the planning process.
• Monitor the development and delivery of member-led training and professional development programs, and modify planned approaches, as necessary.
• Evaluate the effectiveness of the training and professional development provided in terms of its contribution to the achievement of the IRB’s mandate and strategic priorities.

Each of these committees should consist of the DC and the regional ADCs, as well as senior Legal Services and adult learning advisors. The regional ADCs would each be responsible for leading the identification of members’ training needs in their regions, drawing on the results of region-level consultations, legal advisors would provide input regarding the substantive law knowledge required by members, and adult education would advise on the best use of experiential learning and evaluation practices. In the case of RPD, a senior advisor from the Research Directorate would also be beneficial, to provide input regarding training needs related to conditions in countries of origin of refugee claimants.

• Divisional committees should be accountable to the senior leadership of the IRB.
  Each of the divisional committees should be responsible for the development and costing of annual (or biennial) plans for training and professional development, as noted above. These plans should provide an integrated approach to not just the design and delivery of new member training and ongoing professional development but also the systematic use of mentoring and observation or shadowing activities to support the ramping up of new members. The plans and proposed budgets should be presented to the Chair, Executive Director and DCs for review and approval. The committees should also be responsible for providing regular progress reports on the implementation of the approved plans and the results.

• Design and implement a systematic process for identifying member training needs.
  The IRB should establish a consistent process, directed by the national training committees for each division, for identifying member training needs across all divisions and regions. This process should be based on consultations with members and the synthesis of needs identified through member monitoring and performance appraisals. The needs analysis process should be an integral element of the planning undertaken by each of the divisional training and professional development committees.

• Implement a consistent approach to the assignment of mentors for new members and define the scope of the mentor role.
  Mentors are used to varying degree by all divisions; however the use of mentors is not institutionalized across the organization. The period in which mentors provide support to new members will need to be determined on a case-by-case basis that reflects the rate at which new members reach the point where there are considered to be able to make timely and defensible decisions on their own.

• As part of the annual performance evaluation process, ADCs and coordinating members should work with their assigned members to develop personal professional development plans that address identified needs for further training and development.
  According to many of the interviewees the current evaluation process addresses individual training and development needs informally and should be strengthened. Consolidated information from these plans would also provide insights for the identification of training needs as part of the annual planning by the divisional training and ongoing professional development plans (as noted under the previous recommendation).

2. Improve the support for the planning, design and delivery of training programs

• The IRB should ensure that the recommended divisional training and professional development committees have access to senior adult learning advisors experienced in working with professional audiences.
  The role of the adult learning advisors in the divisional training and professional development committees defined in recommendation #1, above, will not to be to teach member training courses (unless they also have relevant subject matter expertise) but to contribute to the identification of training needs, provide guidance on the application of adult learning principles to the design of training programs and courses, and the evaluation of their effectiveness. These principles include such factors as maximizing opportunities to engage participants through the use of experiential learning methods (discussion, simulation, case studies, problem solving), ensuring content is meaningful to the participants, and directly applicable to their work. This expertise may already be resident in LPDD and, if so, suitably skilled advisors should be assigned to support the work of the divisional committees. If not, it will be necessary to secure the needed expertise by contracting services on an as-needed basis or by recruiting suitable personnel.

• Ensure that clerical/administrative support to enable the timely production of training materials is available to those developing and delivering training and that materials used in training are translated in a timely manner.

• Implement an information management process to retain training and professional development materials for follow-up access by members and/or repeat delivery.

3. Evaluate the effectiveness of current adjudicative training for members

Evaluations of the effectiveness of training and professional development approaches and methods provide a critical feedback loop to enable the effectiveness of approaches to be assessed and evidence-based improvements to be made. We recommend the IRB:

• Develop and implement a process for monitoring and evaluating member training to inform the planning work of national planning committees for each division. Key elements to consider include:

• Development and implementation of standard feedback surveys to measure the reactions of participating members following each new member training course as well as structured professional development courses for other members.
• Periodic online surveys of members regarding their satisfaction with the training received, the extent to which it enables them to perform their roles, and to identify possible gaps and weakness.
• Reviews by DCs and ADCs of the extent to which members are meeting both quality and productivity expectations, and the extent to which training programs and courses are contributing to performance in these areas.

There are also a number of areas where immediate evaluations would be beneficial:

• Undertake a formal review of the effectiveness of the re-designed RPD, IAD and RAD new member training to:

• Assess the extent to which it is preparing the participating members to assume their roles.
• Identify areas in need of improvement.
• Assess the suitability of the length and ordering of the various components.
• Assess the feasibility of streamlining the training provided to new members with relevant background experience (similar to what is currently applied in the RAD’s new member training).

• Commission a study to:​

• Identify best practices in utilizing online technologies to provide training and education for professional audiences as a complement to existing face-to-face methods.

• Recommend a suitable approach to the application of online methods to the IRB’s member training and professional development, taking into account current constraints on the ability of IRB’s online video technologies to handle real-time courses involving interactive presentations and discussions involving relatively large numbers of participants in multiple time zones.

4. Addressing unmet training needs

• Divisional training and ongoing professional development committees incorporate plans for developing and delivering enhanced training approaches to better equip members to conduct hearings and make decisions, and manage their caseloads and workflows.

Introduction

This report presents the findings of an Evaluation of Member Training^{Footnote 3} at the Immigration and Refugee Board of Canada (IRB). The focus of the evaluation was on “adjudicative” training provided to IRB members, which consists of training sessions and related professional development activities that aim to improve the effectiveness and/or efficiency of members in fulfilling their role as decision makers. The evaluation thus excluded the examination of other types of “corporate” training provided to members (and to IRB staff), such as training on harassment prevention.

The purpose of the evaluation was to review the design, development, and delivery of adjudicative training services provided to members in terms of their effectiveness, efficiency, best practices, and lessons learned.

The evaluation was conducted by Kelly Sears Consulting Group on behalf of the IRB’s Planning, Evaluation and Performance Measurement Directorate between November 2017 and April 2018.

Overview of the Immigration and Refugee Board of Canada

The mission of the IRB, on behalf of Canadians, is to resolve immigration and refugee cases efficiently, fairly and in accordance with the law.^{Footnote 4}

The IRB carries out its work in three regional offices located in Toronto (Central Region), Montréal (Eastern Region) and Vancouver (Western Region). The Central Region is responsible for Ontario, excluding Ottawa; the Eastern Region for Quebec, Ottawa and the Atlantic provinces; and the Western Region for the western provinces, the Northern services and the Calgary office.

The IRB delivers on its mission through four divisions: Refugee Protection Division (RPD), Refugee Appeal Division (RAD), Immigration Division (ID), and Immigration Appeal Division (IAD). All four divisions hold hearings in the regions and are assisted by registry services and corporate support. Hearings are also held in a small number of itinerant locations. The IRB's National Headquarters are located in Ottawa.

Through the work of each division, the IRB strives to deliver a simpler, more accessible and expeditious form of justice than that provided by the courts. The IRB applies the principles of administrative law, including those of natural justice, in its proceedings. Its decisions are rendered in accordance with the law, including the Canadian Charter of Rights and Freedoms. The IRB is committed to fairness in all aspects of its work. Each case is decided on its own merits by independent decision-makers.

During 2016-17 the IRB delivered on its mandate with a budget of approximately $115M and 976 full-time equivalents (FTEs), including members. Table 1 below summarizes the number of members by division and region, excluding the Deputy Chairs and Assistant Deputy Chairs. The RPD has by far the largest number of members in all regions. Likewise, the majority of members are located in the Central Region.

Table 1 – Number of Members by Division and Region

Note: Each number represents the number of positions, not the number of full-time equivalents (FTEs). (Source: IRB, List of Members (February 2018), www.irb-cisr.gc.ca/Eng/BoaCom/Pages/MemCom.aspx)

IRB Organizational Structure

Members of the IRB are either appointed through the Governor-in-Council (GIC) process, as is the case for RAD and IAD members, or hired under the Public Service Employment Act, as is the case for RPD and ID members and all other IRB employees. All members report through their respective divisional ADCs and DC to the Chair. Legal Services report directly to the Chair. The various core internal services (human resources, labour relations, finance, accommodations, communications, etc.) that support the four divisions report to the Executive Director.

Figure 1, overleaf, illustrates the organizational structure of the IRB.

Figure 1 – IRB Organizational Structure

As noted above, the IRB delivers on its mandate through three regions – Western, Central and Eastern – and through four divisions – RPD, ID, IAD and RAD – supported by a number of administrative support functions. The principal roles of the divisions and the key support functions of relevance to this evaluation study are described below.

1. Refugee Protection Division

   The mandate and role of the RPD are as follows:

• Decides claims for refugee protection;
• Decides applications for vacation of refugee protection;
• Decides applications for cessation of refugee protection; and
• Decides pre-removal risk assessments (not yet in force; to come into force on a day or days fixed by order of the Governor in Council).^{Footnote 5}
2. Refugee Appeal Division

   Launched on December 15, 2012, RAD considers appeals against decisions of the RPD to allow or reject claims for refugee protection. In most cases, no hearing is held as the RAD bases its decision on the documents provided by the parties involved and the RPD record. The legislation sets out who can appeal to the RAD.^{Footnote 6}

3. Immigration Division

   The ID conducts admissibility hearings for certain categories of people believed to be inadmissible to Canada under the law. At the request of the Canada Border Services Agency (CBSA), foreign nationals or permanent residents who are believed to have contravened the Immigration and Refugee Protection Act (IRPA) appear before the ID for admissibility hearings.

   The ID also conducts detention reviews. When the CBSA detains a person and that person is not released, the ID must review the grounds for detention and decide whether there is reason under the IRPA to continue detention.^{Footnote 7}

   A member (decision-maker) will conduct the hearing according to the IRB tribunal process. The process is adversarial given there are two opposing parties: the person concerned and Minister's counsel for the CBSA.

4. Immigration Appeal Division

   The IAD hears appeals on immigration-related matters. This division may allow an appeal and set aside an original decision based on the grounds of an error in law or fact, or of a breach of a principle of natural justice. In certain cases, the IAD may also give special relief on the basis of humanitarian and compassionate considerations, which includes taking into account the best interests of a child directly affected by the decision being appealed to the IAD. In a removal order appeal, the IAD may also give special relief by staying the removal order with conditions. If the IAD does not allow an appeal or stay the removal order made against the appellant, the IAD will dismiss an appeal.^{Footnote 8}

5. Learning and Professional Development Directorate

   The Learning and Professional Development Directorate (LPDD) is part of the Integrated Resources Management Branch. Its mandate related to learning and professional development is to assess, on an ongoing basis, the learning and professional development needs of all IRB personnel, including members, and develop and implement a strategy that will enable the IRB to respond to these needs in a timely, relevant and integrated manner. As outlined later in this report, LPDD’s role in supporting member training varies widely across the divisions.

   In 2015-16 the LPDD’s mandate was expanded to include other responsibilities such as Performance Management, Performance Pay, Talent Management, Awards and Recognition, Official Languages, Employment Equity, and Surveys such as the Public Service Employee Survey.
   

6. Legal Services

   Legal Services is responsible for providing legal and policy advice to the Chairperson, senior management, decision-makers in all divisions and other IRB personnel. As one of its activities, upon request, Legal Services also reviews draft reasons for decision. However, members are free to accept or reject the advice provided by Legal Services.

   The role of Legal Services in providing member training varies by division and region. However, in all cases, Legal Services is responsible for providing members with periodic updates on legal decisions and their implications for the IRB, referred to as “jurisprudential updates”. In addition, Legal Services plays a key role in designing and delivering new member training, with some member participation.

Evaluation Methodology

The evaluation covered the adjudicative training provided to new and existing members over the period from March 31, 2013 to December 2017.

Prior to the evaluation a separate study was undertaken to develop a research design for the evaluation.^{Footnote 9} That planning study prepared a research matrix, proposed the data collection methods, and developed the associated data collection instruments. The primary methods proposed were: key informant interviews; an online survey of members; and a comparative analysis of member training at two other organizations. The research matrix, i.e., the list of evaluation issues/questions and the corresponding measurement indicators and data sources, is provided in Appendix A.

During the planning phase of the evaluation, Kelly Sears worked with the IRB Planning, Evaluation and Performance Measurement Directorate to finalize the methodology and refine the data collection instruments. A summary of the evaluation methods is provided below.

1. Key Informant Interviews

   A total of 41 in-person and telephone interviews were undertaken with IRB representatives, covering Headquarters and all of the divisions and regions. The breakdown of interviews by organizational unit is as follows:

• Executive Office (n=1).
• Learning and Professional Development Directorate (n=6).
• Legal Services (n=9).
• Immigration Division (n=12).
• Immigration Appeal Division (n=5).
• Refugee Protection Division (n=6).
• Refugee Appeal Division (n=2).

The interview guide is provided in Appendix B.

2. Online Survey of Members

   The online survey of members was intended to answer several study questions related to the structure and effectiveness of member training. The survey questionnaire is provided in Appendix C.

   The survey was conducted over a two-week period between January 24th and February 6th, 2018. The survey targeted all members in all of the regions of the four adjudicative divisions. A total of 241 IRB members^{Footnote 10} were sent an invitation to participate in the survey, followed by two reminder emails to non-respondents as the survey progressed. A total of 113 survey responses were received, for a response rate of 47%, which is higher than the typical response rates for internal surveys reported by online survey companies.

   The breakdown of response rates by division, and separately, by region, is shown in Table 2. The response rate within the RPD, which accounted for 57% of the total responses, was very similar to the overall rate. Rates within the RAD, ID and IAD were broadly similar to the overall rate, recognizing that small differences in the number of responses can have a greater impact on the percentages. Response rates by region were also broadly in line with the overall rate. The small numbers of respondents from the RAD, ID and IAD mean that results for these divisions are highly sensitive to small variations in the numbers of responses, for example, each ID response represented almost 8% of the total for the division. Consequently, the analysis and presentation of survey findings is largely focused on results across all four divisions. Results for the RPD (for which the sub-sample was large enough to be separately analysed are noted where they differ from the overall results. Findings for the RAD, ID and IAD are noted where there are marked differences although such differences should be considered indicative only.

   The survey findings were presented to the IRB, including the response rates by division and region, in the first draft evaluation report. The IRB Project Authority accepted the response rates as valid and is confident in the survey findings presented in this report.

   Table 2: Survey Response Rates among IRB Members Contacted

   By Division
   Division	# of Responses	# Contacted	%
   RPD	64	141	45%
   RAD	18	31	58%
   ID	13	34	38%
   IAD	18	35	51%
   Total	113	241	47%

   By Region
   Region	# of Responses	# Contacted	%
   Eastern	34	65	52%
   Western	62	142	44%
   Central	17	34	50%
   Total	113	241	47%

   With regard to length of service of the members who responded, 45% became members in their current divisions in 2016 or 2017, 34% between 2011 and 2015, and 21% prior to 2011. RPD, RAD and IAD distributions were similar to the overall pattern, whereas a large majority of ID respondents (77%) were appointed to their current positions prior to 2011.

3. Comparative Analysis

   A comparative analysis examined alternative approaches and best practices relating to training and professional development in other similar organizations. Two organizations were selected for the comparative analysis – the National Judicial Institute (NJI) and the Social Security Tribunal (SST). The comparative analysis involved the conduct of an in-person interview with a senior manager knowledgeable about adjudicative training in each organization, along with a review of documentation and literature available online. The interview guide for the interviews with representatives of the two comparative organizations is included in Appendix D.

4. Review of Documents and Data

   A review of documentation relating to the IRB and member training was undertaken to help address all study issues and questions. The documents reviewed included annual departmental plans and results reports, sample training materials, and previous studies (particularly an organizational review of the IRB Learning and Professional Development Directorate in 2005).

   Limited data was available related to such items as the number of training sessions offered by each division, the time and costs devoted to the design and delivery of training or on the performance of member training. The IRB’s Planning, Evaluation and Performance Measurement Directorate provided the evaluation team with time series data on member productivity in order to help assess whether the revamped approach to new member training (in RPD) enables new members to become productive more quickly (evaluation question #5 in the research matrix in Appendix A).

5. Analysis and Presentation of Findings

   The two primary sources of information for this evaluation come from the key informant interviews and the survey of members. The data from the key informant interviews is, to the extent possible and appropriate, presented at the level of each division and region. Where there are variations in how IRB member training and professional development is planned, designed or delivered, the key informant interview findings are presented so that these variations are evident. This is possible because the interview findings are qualitative and interviewees were probed during the interviews to explain their perspectives. Survey results are quantitative and the number of responses by division and region are generally not sufficient to allow detailed analysis by division and region. However, where possible survey results are presented by region or division.

Findings

Overview of Member Training and Professional Development at the IRB

As outlined in section I, the focus of the evaluation study was on adjudicative training provided to IRB members. Adjudicative training consists of training courses and related professional development activities that aim to improve the effectiveness and/or efficiency of members in fulfilling their role as decision makers. The evaluation thus did not examine other types of corporate and workplace training provided to members or to IRB staff (e.g., training on harassment prevention or values and ethics).

Members are provided with training beginning with their initial appointment as new members and on an ongoing basis throughout their terms of appointment. For the purposes of the evaluation, adjudicative training was grouped into four categories:

• Orientation and training for new members – All new members, including both GIC appointees and public servants, undergo new member training specific to the division to which they have been appointed/hired.
• Ongoing training for current members relating to the substantive law and knowledge required for decision-making – Legal Services monitors Federal Court decisions that may have implications for the decision-making of the IRB. Members receive regular updates, often monthly, on recent court decisions and their jurisprudential implications (referred to as jurisprudential updates). These briefings are usually provided in the form of presentations during monthly professional development sessions (“PD Days”) provided to members in each division and region.
• Ongoing training for current members relating to skills in conducting hearings and making decisions (e.g., reviewing evidence, writing decisions).
• Ongoing training for current members relating to skills in managing caseloads and workflows (e.g., case management, time management).

The identification of member training needs and the development and delivery of training services is mainly the responsibility of two groups: the four divisions (they both define training needs and may be involved in the design and delivery of training sessions) and Legal Services (subject-matter experts who also help to identify member needs and develop and deliver the training). The Learning and Professional Development Directorate has had limited involvement in member training in recent years.

The IRB currently does not have a single system to track information on all of the training provided to members across the country. During the evaluation study each division was asked to provide an inventory of training provided.

Roles and Responsibilities and Accountability

This section describes the roles and responsibilities with respect to member training and professional development, followed by an assessment of accountability for these activities.

1. Roles and Responsibilities

   Summary: In all divisions, for all regions, it is the DCs and/or the ADC who has the primary responsibility for identifying and responding to member needs. In this sense, the DCs and ADCs view themselves as the primary clients of Legal Services who develop and deliver the majority of the training and professional development.
   

   The level of collaboration between Legal Services, whose lawyers are the legal subject-matter experts, and LPDD, which traditionally has had a mandate for training (to members and staff), has been almost non-existent in recent years. LPDD representatives indicated that the directorate’s resources have been decreased to the point where they no longer can provide training services to members.

   LPDD has generally not been involved in the design and delivery of new member training and professional development for most divisions. IAD has made use of LPDD in the design and delivery of new member training and RAD approached LPDD for assistance but LPDD were unable to assist due to limited resources. RPD and ID have chosen not to make use of LPDD in the design and delivery of new member training and professional development. This reportedly stems from negative experiences in the past. Another factor may be geographic distance: LPDD’s resources are centred at NHQ in Ottawa while most IRB members are located elsewhere, with a concentration in the Central Region (Toronto).

   At the level of the divisions, each division tends to design and deliver its own training, i.e., there is no collaboration across divisions. In fact, there is evidence of only limited, ad hoc collaboration within divisions. The approach to the design and delivery of member training within the IRB was described as “siloed” by interviewees.

   Most interviewees believe that the roles and responsibilities across the organization with respect to member training and professional development are not clear. Specifically, the role of LPDD is seen as being somewhat ambiguous because it has been involved in training for IAD and providing clerical support for some divisions in some regions. The relationship between LPDD and Legal Services in designing and delivering training and professional development to members is also seen as lacking in clarity. Some, however, did state that the role of LPDD is clear – they provide mandatory training for public servants and serve an administrative/clerical role when divisions and/or Legal Services deliver training.

   One of the central issues of the evaluation study was to assess the relationships between the various organizational units involved in the design and delivery of member training services. More specifically, senior management was interested in determining the appropriate role, if any, for LPDD and for Legal Services in developing and providing member training services.

   In most divisions, interviewees reported that LPDD has been largely isolated over the past few years, with most divisions opting to not use the services of LPDD in planning, designing or delivering training to their members. The unwillingness to use LPDD stems, according to interviewees, from some negative experiences in the past, particularly the new member training delivered within RPD in 2012. In that year, reforms at the IRB resulted in an influx of a large number of new members who needed to be trained in a short period of time. LPDD was given three months to develop new member training for approximately 100 new RPD members. According to interviewees this training did not meet the needs of members, and LPDD has never been able to fully recover from this experience.

   A further challenge in the collaboration with LPDD in planning, designing and delivering training is that LPDD’s resources are concentrated in Ottawa at NHQ. LPDD has limited presence (only one Learning Advisor) in the Central Region where the largest numbers of members are located (almost 60% of members are located in this region). Overall, the current relationship between most divisions and LPDD was described as “antagonistic”.

   Furthermore, there is reported to be a lack of collaboration between Legal Services and LPDD. Interviewees indicated that Legal Services has in the recent past asked LPDD for assistance in planning, designing or delivering training to members but LPDD has indicated that they lack sufficient resources to provide the required assistance. Other interviewees report that in some instances Legal Services has been unwilling to collaborate with LPDD.

   At the division level, interviewees generally believe that there is a lack of collaboration across divisions when it comes to member training. Training is fragmented, within each division, and often within each region in a division, focusing on its own training needs. However, there is a sense that there are common needs across divisions and across regions and thus increased collaboration would be beneficial in terms of efficiency and effectiveness of member training and professional development.

   Most interviewees believe that the roles and responsibilities related to the planning, development and delivery of member training are not clear or well understood. The role of LPDD is the main source of confusion or lack of clarity noted by interviewees – in the past LPDD has been involved in training for some divisions and regions but not for others. The relationship and interaction between LPDD and Legal Services is also unclear to many interviewees. In fact, some interviewees noted that LPDD is no longer involved in member training yet, based on interviewee comments, not everyone is aware of this.

   Those who indicated the roles and responsibilities are clear generally commented that there is not much role for LPDD in member training, beyond mandatory training for public servants (e.g., harassment-free workplace training) and an administrative/clerical role, such as booking conference facilities, printing materials, making travel arrangements, etc. These interviewees suggested that the mandatory training currently provided by LPDD might possibly be contracted out to consultants, thereby eliminating the need for LPDD to provide this type of training.

   There is some evidence of tension related to the role of Legal Services in providing member training and professional development. As explained by some interviewees, Legal Services are advisors, not decision-makers, and as a result Legal Services do not always view things from the same perspectives used by members, and sometimes have views that do not align with those of senior management. The subject matter for new member training as well as some ongoing professional development is often very legalistic and complex and thus Legal Services are required to develop the content for the training modules. However, according to interviewees the situation has evolved so that Legal Services both develops and delivers the content and thus are able to shape the culture of decision-making at the IRB. A number of interviewees questioned the appropriateness of the balance between the respective perspectives of Legal Services and members with regard to best approaches to decision-making practices.

   There are variations across divisions with respect to the roles and responsibilities related to training and professional development. Findings for each of the four divisions are provided in turn below.

   1. Immigration Division

      According to interviewees familiar with training in the ID, the division has chosen not to collaborate or make use of LPDD for a number of years. They explained that LPDD is not relevant to the training needs of the division because LPDD does not have the legal background required to plan, design or deliver the training. As explained by a few interviewees, “they cannot answer questions related to the law.” The Eastern Region does make use of LPDD for administrative/clerical tasks associated with training.

      The ID uses Legal Services extensively to plan, design and deliver training to its members. For the most part, the relationship is reported to work well. However, one region expressed frustration over the way jurisprudential updates were being delivered, noting that both the substance and delivery methods were not meeting the needs of the region. Despite attempting to discuss the issue with Legal Services, regional representatives have not been successful at getting Legal Services in this region to modify their approach. This is in stark contrast to another ID region which expressed great satisfaction with the jurisprudential updates delivered by Legal Services.

      With respect to roles and responsibilities, interviewees reported that the division takes the lead in member training, i.e., ID is the client for training, and Legal Services is the service provider. Legal Services works closely with the DC and with the ADC in each region to identify member training needs and to deliver training that reflects these needs. Legal Services plays a key role in identifying training needs based on jurisprudence and in reviewing member’s reasons for decisions.

   2. Immigration Appeal Division

      In contrast to the ID, the collaboration with LPDD in the IAD was reported to be positive. Notably, the IAD recently used the skills of LPDD to revise the new member training course. IAD is reported to have been very satisfied with the services provided by LPDD. Some interviewees from IAD noted that although LPDD does not have the substantive knowledge related to the work of the IAD, they are seen as having the skills and knowledge related to adult learning principles and methods.

      The IAD uses Legal Services for knowledge-based material while LPDD is seen as having a role in identifying techniques and suggesting ways to deliver the training. Individual IAD members are seen as having a role in training as well in sharing their knowledge and experience.

   3. Refugee Protection Division

      Interviewees from RPD generally believe that they cannot make use of LPDD’s services because the directorate does not possess the substantive legal knowledge required to plan, develop and deliver training to its members. There is no evidence of any collaboration between LPDD and RPD, particularly in the Central Region where most RPD members are located and where the new member training was re-designed based on experiential learning methods. In revising its new member training, the Central Region opted to make use of external consultants from St. Francis Xavier University. Regional RPD management believe that LPDD did not have the skills and substantive knowledge to undertake this substantial project, while LPDD indicated that they could have met the needs of RPD in redesigning its new member training.

      Interviewees indicated there is limited collaboration among the RPD regions. Training is usually regional (there is no national training committee) and when there is some national training, it is dependent on one of the ADCs to take the initiative and contact his or her counterparts in the other regions. One interviewee described the process as “siloed” and “ad hoc”. Interviewees expressed the view that there is a need for much more coordination, which could be accomplished by establishing a national committee that would meet a few times per year to discuss training. However, there is also belief that there will always be a need for a regional approach because the realities of each region differ somewhat (e.g., claim types differ).

      In RPD, the division is seen as the client with Legal Services delivering much of the training. Legal Services is seen as being well placed to provide input into the training because it is viewed as having the subject-matter experts in the law. However, a few interviewees noted that Legal Services has no expertise in pedagogy and thus they should not have a strong role in designing and delivering training to members, i.e., the role of Legal Services should be focused on providing its subject matter expertise.

   4. Refugee Appeal Division

      LPDD is not involved in training within RAD and thus was not involved in the redesign of the new member training course. Collaboration between RAD and Legal Services is reported to be good in all regions. Legal Services works with the professional development committees in the regions to identify topics for training and often provides the training as well.

      With respect to roles and responsibilities, interviewees report that these are largely clear. The division is seen as the lead in member training, i.e., RAD is the client for training and Legal Services is the service provider. Legal Services works closely with the ADC responsible for the RAD training strategy and the DC to identify member training needs and to design and deliver training that reflects these needs.

2. Accountability for Training

   Summary: There was agreement that the ADCs in each region and the DCs at the national level are accountable for ensuring members receive the right training at the right time. However, there is a lack of national coordination within divisions and across divisions. 

   There was general agreement that the ADCs in each region and ultimately the DC of each division is accountable for ensuring members receive the right training at the right time. A few interviewees noted that the IAD and RAD have relatively fewer members and thus the ADCs in each region are well-placed to closely monitor the performance of members and identify individual member training needs. Most interviewees stated that this approach works well from an accountability perspective, as those who review member performance are also responsible for identifying member training needs and ensuring training is provided when needed.

   Some interviewees questioned the impact on consistency of training, and thus potentially decision-making, across regions when accountability rests largely in the region. These interviewees argued that this decentralized approach may be resulting in members in different regions receiving different training.

Approaches to planning and delivery

1. Identification of member training needs

   Summary: Overall, the identification of member training needs is a collaborative process between each of the four divisions and Legal Services.

   The process followed to identify member training needs differs slightly across divisions and, in some divisions, across the regions. However, in all cases, the divisions take the lead in identifying training needs for members. For new member training, the needs are seen by interviewees as being well-established: all new members receive training that is specific to the division. However, the training is not necessarily identical in all regions within each division.

   The IRB does not have a national professional development committee which would identify the training needs common across divisions and coordinate the provision of training nationally. The existence of national professional development committees within the divisions has varied over the years, influenced by the style of the senior managers (the Chair, DCs and ADCs) of the day.

   With respect to the provision of jurisprudential updates, Legal Services reviews legal decisions from across Canada and identifies whether training is required and discusses this with the ADC in each region. Legal Services delivers a summary of the legal decisions and implications for each division in each region.

   A key challenge identified by some interviewees with respect to ongoing training is the differing training needs of members – making it difficult to plan training that is relevant to all members. This is particularly challenging in regions with fewer members.

   Two-thirds of members surveyed indicated they were satisfied with the needs identification process, indicating there is room for improvement. Suggestions for improvements mostly related to systematically soliciting member input and establishing individual training plans as part of the members’ performance reviews. 

   The next four sections summarize how training needs for ongoing professional development are identified in the divisions. We then summarize the level of satisfaction of members with the identification of training needs based on the survey findings.

   1. Immigration Division

      ID management decides on a direction for training for the coming year, including any modifications required to the new member training course. In addition, ID management meetings involving the DC, ADCs and Legal Services are held every two weeks and training is often part of the agenda. Legal Services provides input on training needs based on jurisprudence and reviews of member decisions. Interviewees reported that ID management seeks to coordinate member training nationally to the extent possible in order to ensure consistency across regions.

      In recent years the ID had no formal national professional development committee until October 2017 when a new committee was struck, with member representation from all three regions. The role of the committee includes identification of members’ training needs and determination of best approaches to the design and delivery of the needed training while taking variations in regional needs into account.

      At the regional level, training needs are identified through a variety of means. Regional managers hold discussions with individual members about their training needs; a sample of member decisions are reviewed to identify areas for improvement; and, the results of individual performance reviews flag issues that need to be addressed.

   2. Immigration Appeal Division

      The IAD has a national professional development committee that includes members as well as representatives from Legal Services. This PD committee recently has become more active under the new IAD DC. In addition, there are regional professional development committees which include members and a representative from Legal Services.

      Ongoing training needs are identified largely independently in each region via regional professional development committees that include members and representation from Legal Services as well as more informal observation by ADCs and coordinating members of the performance of members and insights from individual performance evaluations.

   3. Refugee Protection Division

      In the RPD, there is a professional development committee in each region which includes representation from members and Legal Services. A national professional development committee previously existed but now is reported to be defunct. Training needs are identified in the regions through reviews of member decisions, member performance reviews, managers attending member hearings, and communication with members who identify training needs. In the Eastern Region, members are periodically sent a survey by regional managers asking about their training needs.

   4. Refugee Appeal Division

      The RAD established an Adjudicative Strategy Committee in the summer of 2017 whose mandate includes providing direction for the professional development in the division. The committee is chaired by the ADC Training (a special ADC position created in early 2017 to oversee the design and delivery of training for RAD in anticipation of an influx of new members) and has member representation from each of the regions as well as Legal Services. The committee initially focused on planning the division’s 2017 national meeting and has now taken on the selection of topics to be covered division-wide in monthly PD day sessions. These PD sessions are complemented by regular member meetings (usually every two weeks) that are held in each region that address topics of common concern or interest relating to members’ roles.

   5. Member satisfaction with the identification of training needs

      Members who completed the online survey were asked three questions about the identification of training needs by their divisions: satisfaction with the process, extent to which they are consulted as part of the division’s needs identification process, and suggestions as to how divisional processes may be improved. The results show that two-thirds (66%) of the members who responded were completely or mostly satisfied with the way training needs are identified, and a similar proportion feel they have been consulted as part of the process. Only a small proportion (8%) indicated they were dissatisfied with the needs identification process or felt they had not been adequately consulted (14%). Suggestions for improvements mostly related to systematically soliciting member input and establishing individual training plans as part of the members’ performance reviews.

      As shown in Figure 2 the satisfaction levels were reasonably similar across the four divisions. Small proportions in RPD (8%), RAD (11%) and ID (15%) indicated they were dissatisfied with the needs identification process, and no IAD members were dissatisfied. (Note that the results for RAD, ID and IAD are based on small sub-sample sizes and thus should be viewed as indicative only.)

      Figure 2: Level of satisfaction with identification of members' training needs

      

      (Note: Small sub-sample sizes: RPD-64, RAD-18, ID-13, IAD-18.)
      

      • Text format

        This is a bar chart that describes how satisfied members of the RPD, RAD, ID, and IAD were with the identification of their training needs.

        The first bar depicts the RPD satisfaction percentages. 64% were completely or mostly satisfied, 25% were neither satisfied nor dissatisfied, 8% were completely or mostly dissatisfied, and 3% were not aware the division identified needs.

        The second bar depicts the RAD satisfaction percentages. 72% were completely or mostly satisfied, 17% were neither satisfied nor dissatisfied, and 11% were completely or mostly dissatisfied.

        The third bar depicts ID satisfaction percentages. 62% were completely or mostly satisfied, 15% were neither satisfied nor dissatisfied, 15% were completely or mostly dissatisfied, and 8% were not aware the division identified needs.

        The fourth bar depicts IAD satisfaction percentages. 72% were completely or mostly satisfied, 22% were neither satisfied nor dissatisfied, and 6% were not aware the division identified needs.

        The fifth bar depicts total satisfaction percentages. 66% were completely or mostly satisfied, 22% were neither satisfied nor dissatisfied, 8% were completely or mostly dissatisfied, and 4% were not aware the division identified needs.

      The results from the follow-up question asking respondents to indicate the extent to which they had been consulted as part of their division’s needs identification process are shown below.

      Level of Consultation re: Training Needs	% of Respondents
      To a great extent	20%
      To some extent	40%
      To a limited extent	25%
      Not at all	14%
      Don’t know / Not sure	1%
      Total	100%

      These data show that 60% felt they had been consulted to at least some extent, and only 14% felt they had not been consulted at all. Members from the ID and IAD were slightly more likely to indicate that they had not been consulted at all, although the numbers involved were very small due to the sub-sample sizes for these two divisions. Members appointed prior to 2015 were more likely to feel they had not been consulted compared to the more recent appointees (17% vs. 12%).

      Suggestions as to how divisions could improve their processes for identifying members’ training needs mostly related to:

      • Introducing more systematic and frequent consultations with members regarding training needs.
      • Incorporation of individual training objectives into members’ performance evaluations.
      • Collection of feedback on the effectiveness and value of training sessions following the completion of such sessions, either immediately after or after some time has elapsed and recipients have had time to reflect on the value of the training received.
      • Give priority to member requests/needs over judgments by Legal Services regarding training needs and content.

      Suggested methods for consultation with members ranged from training discussions/suggestions during member meetings and PD days; use of email requests, focus groups and surveys to solicit member input; and, mandating the Adjudicative Strategy Committee and PD committees to consult and solicit training suggestions.

2. Planning, Design and Development of Training

   Summary: The approach to planning and designing of training varies by region and division. In general, the ADC or DC of each division has final say on the agenda for each training session, including new member training as well as ongoing professional development. There is agreement that the design ongoing training in all regions is done largely by Legal Services with assistance/input from members. Re-design of new member training undertaken by RPD, RAD and IAD was undertaken by ACs/ADCs and members with support from Legal Services. Limited training is planned or designed in collaboration with LPDD. 
   

   1. Immigration Division

      The ID ADCs coordinate the new member training as well as ongoing training, consisting of PD days and one-off training sessions requested by members. Once the ADCs and DC decide on the need for a training module, with input from Legal Services, the development of it rests with Legal Services in the regions because the content mostly is legal in nature.

   2. Immigration Appeal Division

      In the IAD, new member training is now standardized following the re-design of the course in 2017 and, as of February 2018, is being translated for implementation in the Eastern Region. The new course was designed at National Headquarters collaboratively by the DC and LPDD. The new curriculum is, according to interviewees, less judicial and emphasizes the difference between administrative justice and judicial justice. The new training curriculum has greater focus on practical skills and the approach is member-led whereas the previous version was led largely by Legal Services. The new member training course is three weeks in duration. The training makes use of PowerPoint slides and case studies as the main teaching tools. For the monthly PD days, the regional professional development committees, which include a Legal Services representative and members, are responsible for the planning and design of training with the ADCs responsible for the approval of training topics.

   3. Refugee Protection Division

      The planning and design of training in the RPD is regionally based. A major initiative has been the revamping of the new member training course. RPD in the Central Region, with strong support from Legal Services, took the lead in developing a totally new approach based on experiential learning principles, referred to as “ERGA” (experience – reflection – generalization – application). The new member training at RPD is ten weeks in duration, with eight weeks of class time and two weeks of practical training. The training is broadly broken down into three sections: hearing preparation with an emphasis on the law, presiding over hearings, and making decisions. The new course has been held twice in Central Region (2016 and 2017) with a third wave scheduled for 2018. Some interviewees noted that new member training in the RPD is not consistent across regions with the Central Region delivering the new ERGA-based training developed in the region while the Eastern and Central Regions provide new member training planned and designed in each region.

      Ongoing training to members is planned and designed in the regions by the regional professional development committees with the ADC in each region having final sign-off. In the regions, Legal Services handles all legal aspects based on the identified needs. Interviewees noted that there is a significant amount of interaction between Legal Services and members in the planning and design of training. For nationally-based training, design is sometimes done in Ottawa by Legal Services.

   4. Refugee Appeal Division

      New member training in RAD was redesigned in 2017 with implementation beginning in May of that year. The RAD revised its new member training in 2017, with changes focused largely on the way the information was delivered rather than changes to content. The first cohort of new members to receive the revised training was in August 2017 (three new members) and a second cohort in September 2017 (two new members). The redesign was led by a RAD member with assistance from a team of three lawyers in Legal Services; the same team also delivered the training sessions in August and September.

      The approach to ongoing training and professional development is collaborative, working through the division’s Adjudicative strategy and professional Development Committee, which has representation from members in the three regions and Legal Services. Legal services leads most of the design and delivery of this training, working in response to the directions set by the committee. Legal Services also prepares and presents monthly jurisprudential updates on case law and legal topics applicable to the work of the RAD as a core element in the monthly professional development sessions.

3. Delivery of Training

   Summary: The approach to new member training varies by division and by region with all divisions except ID having developed formal adult-learning focused approaches to new member training in recent years. The revamped approaches to new member training usually require more initial time for training new members; however this is seen by some interviewees as being offset by new members becoming fully productive more quickly. (An analysis of data on trends in member productivity is presented later in this report.)

   In general the approach to ongoing training has not changed significantly to reflect adult-learning approaches. On-going training is delivered almost exclusively by Legal Services in all divisions and all regions. 
   

   1. Immigration Division

      Most of the new member training is delivered by Legal Services in the regions with some input from members who provide training on hearing room management. For the ID, a challenge for new member training is that often only one or two members are appointed at a time, making it impossible to benefit from economies of scale or for new members to benefit from interaction with other new members.

      National training seminars are delivered by the ID three to four times per year via video conference, with in-person national training seminars taking place every second year.

      Monthly PD days on case law/jurisprudential updates are delivered by Legal Services in the regions.

   2. Immigration Appeal Division

      National PD days are delivered monthly, except during the summer months. Each PD session is a full day with a half-day focused on training and another half-day allocated to members meeting with each other.

   3. Refugee Protection Division

      As outlined above, the redesigned new member training course based on ERGA principles has been held twice in Central Region with a third cohort to be trained in the spring of 2018. It is co-led by a member and a representative from Legal Services. In the Eastern and Western Regions, the new member training is three to five weeks in duration and is delivered by two members and a Legal Services advisor.

      For monthly PD days, topics are proposed by the professional development committee and often include external speakers and sometimes focus on specific needs such as decision writing. Jurisprudential updates are delivered by Legal Services and country discussions are generally led by the Research Group.

   4. Refugee Appeal Division

      In RAD, new member training is delivered by Legal Services and members. Other professional development training – monthly professional development days plus periodic national meetings – within the RAD is delivered by members, guest speakers and Legal Services.

      An important distinction for RAD is that many new members do not have experience with refugee protection legislation and so, because of the nature of the work, RAD members must receive training related to RPD as well as RAD in order that they be equipped to make decisions related to RPD appeals. As a result, new member training in the RAD is more variable across members based on their previous experience.

4. Monitoring and Evaluation

   Summary: There is no formal, consistent process to monitor and report on the effectiveness of member training at the IRB.
   

   There was agreement among interviewees that there is no formal process for monitoring and evaluating member training services or the outcomes of training at the IRB.

   A few interviewees indicated that sometimes debriefs or feedback sessions are conducted with members, usually following new member training. For example, interviewees noted that IAD and RPD sought member feedback following the recently implemented new member training sessions as part of a continuous improvement process. Feedback from members who had completed the revised new member training was used to improve the training for the next cohort of new members.

   The ID Eastern Region indicated that there is follow-up with new members six months post training to obtain feedback on how well the training received reflected their needs and whether any changes are required to the new member training regime.

   One (but not the only) measure of the efficacy of new member training is the time taken by members to reach full productivity, measured as the number of decisions produced per month. The IRB’s Planning, Evaluation and Performance Measurement Directorate has conducted such an analysis, which is presented in the following section.

Effectiveness of Training and Professional Development Activities

This section addresses the effectiveness of training in meeting the learning needs of IRB members. The findings in this section are based largely on the perceptions of interviewees and results from the survey of members.

1. Extent to which Training Equips Members for their Role

   Summary: The new approaches to new member training are seen as an improvement to the previous models. The new approaches are viewed as more reflective of new members’ previous experience and adult learning needs. However, the increased time and resources required to train new members using these new approaches are concerns to some. An analysis of trends in new member productivity in the RPD Central Region indicates the impact of the new approach on productivity is modest, although it is too early to draw a definitive conclusion.

   Although the ongoing training related to substantive law is seen as meeting the needs of members, ongoing training related to conducting hearings and making decisions and managing caseloads and workflows is not entirely meeting the needs of members. This perspective is supported by data from the survey of members.

   The most commonly undertaken training among the surveyed members was that relating to ongoing substantive law and knowledge for decision-making (87% of respondents). Training on conducting hearings and making decisions was undertaken by 50%, new member training by 46% – primarily by members appointed in 2016 or 2017 – and training on managing caseloads and workflows by 21%. Large majorities of the participants in each type of training – from 63% to 92%, depending on the type of training – were either completely or mostly satisfied with the training they received.

   Ratings of the extent to which the training received enabled or facilitated the achievement of expected outcomes show that large majorities of members surveyed completely or mostly agreed that the different types of training enable members to make well-reasoned and timely decisions, and provide immediately applicable knowledge and guidance. However, minorities of the responding members felt that the new member training and ongoing training on substantive law and conducting hearings enable them to better manage their caseloads and workflows, and only a small majority of those who received training on managing caseloads found it to be useful.

   Findings from the key informant interviews related to the four categories of training and professional development are presented in turn in below, followed by the findings from the survey of members.

   1. New Member Training

      As noted in previous sections, the IRB is moving towards revamping its new member training programs based on adult education principles, known as “experiential learning”. RPD in Central Region took the lead in developing the new approach, followed by IAD and RAD. As of the time of our research in early 2018, the ID had decided not to adopt the new approach “experiential leaning” approach. However, Legal Services is currently revising the methods of new member training in consultation with the DC and ADCs.

      Those familiar with the new approaches indicated that they are more reflective of new members’ previous experience. For example, in RAD, the revised approach has two levels. The “off the street” members who know very little about refugee law and tribunal adjudication receive the complete training, including a condensed version of the RPD training which is delivered over about four weeks (regular RPD new member training is eight to ten weeks in length). They then receive RAD-specific training, which takes approximately three weeks. New RAD members with some previous background in refugee law receive only the RAD-specific component of the training as well as some supplementary training if needed.

      The ERGA approach implemented by RPD and the approaches implemented by IAD and RAD are based on adult learning principles. Interviewees report that feedback received from members in RPD, IAD and RAD indicates that they are very satisfied with the new approach. Some managers also believe the new approach has enabled members to become fully productive sooner than was the case under the previous training regime.

      In order to test the hypothesis that the ERGA-based training enables new members to achieve full productivity in a shorter period of time, the IRB’s Planning, Evaluation and Performance Measurement Directorate conducted an analysis of trends in member productivity in RPD Central Region. Figure 3 presents a chart of member productivity (defined as the number of full hearing decisions per year) by months of experience.

      Figure 3: Member Cohort Productivity by Months of Experience, RPD Central Region

      

      (Base: Survey respondents who selected ‘too little’ or ‘too much’ training, and provided suggestions regarding the relevant subject areas)
      

      • Text format

        This is a scatterplot describing trends in RPD member cohort productivity by months of experience in the central region. The vertical axis measures the number of full hearing decisions made per year and the horizontal axis measures a member’s months of experience. Two trends are displayed differentiated by two cohorts: members who began working between 2012 and 2014 and members who began working between 2015 and 2017. The 2012-2014 cohort decided approximately 75 hearings per year with 12 months of experience while the 2015-2017 cohort decided approximately 85 hearings per year with 12 months of experience. The 2012-2014 cohort decided approximately 100 hearings per year with 24 months of experience while the 2015-2017 cohort decided approximately 115 hearings per year with 24 months of experience. The sample size for the 2015-2017 cohort is much smaller than the 2012-2014 cohort for this measure and therefore results must be interpreted with caution. The 2012-2014 cohort decided approximately 85 hearings per year with 36 months of experience. Data for the 2015-2017 cohort tapers off after 24 months and is not yet available.

      The chart displays the productivity of two new member cohorts: one group started during 2012 to 2014 and a second group started during 2015 to 2017. Productivity of the first group appears to level off after about 24 months, which jives with the views of key informants, i.e., following new member training it can take up to two years for a member to reach full productivity.

      As noted above, the redesigned ERGA-based new member training began to be introduced in 2016 (one wave in 2016 and a second wave in 2017). While it is premature to draw any definitive conclusions, it appears that members in RPD receiving the new ERGA-based training may be achieving slightly greater productivity in a shorter period of time.

      Some interviewees were concerned that the new approach takes much more time than the previous approach, both on the part of those designing and delivering the training and those receiving it. The general view is that the increased level of investment is acceptable if new members are able to become fully productive in a shorter period of time. Based on the data in Figure 3, – where the 2015-2017 cohort achieved an average rate of 100 full hearing decisions per year one month sooner than the 2012-2014 cohort – it appears that the greater investment in new member training may be warranted if new members ramp up more quickly.

      Readers should note that while new member training and ongoing professional development are critical to equipping members with the knowledge required to render decisions, many other factors, including member competencies and aptitudes for independently managing hearings, making decisions and documenting those decisions also play a critical role in affecting productivity. The effects of these two other determinants cannot be readily separated in the analysis shown in Figure 3.

      The new approach to training may not meet the needs of all members in all regions because some regions may train only a very few members (i.e., one or two) at a time and thus some interviewees wondered if the ERGA-style method could be used in these situations. A representative of Legal Services in the Central Region who has been involved in the design and delivery of the new course indicated that that the ERGA-style approach can be used for groups as small as two members.

      One option suggested by a few interviewees would be to bring new members from smaller regions (e.g., Western Region) to Toronto for training, as currently happens for some new member training in RAD and IAD However, this is not seen as a viable option because of the cost of having new members stay in Toronto for the six to eight weeks required for training. Another issue is that new GIC members can be appointed and start work at any time. A new member appointed in, say, January in the Western Region would have to wait until the next scheduled course in the Central Region, which could be several months later.

   2. Ongoing Training Related to Substantive Law

      There was general agreement among most interviewees that the jurisprudential updates, which are provided by Legal Services via monthly updates, as part of PD days or at national training seminars, equip members with the necessary knowledge related to substantive law.

   3. Ongoing Training Related to Conducting Hearings and Making Decisions

      Evidence from the interviews indicates that training related to conducting hearings and making decisions is not a significant part of the ongoing training provided to members. In general, interviewees indicated that members receive training in reviewing evidence and writing decisions when the need is identified either by the members themselves or by their managers during performance reviews. In cases when members need or ask for training related to conducting hearings or making decisions, the Society of Adjudicators and Regulators (SOAR) was noted as a source of this training, which has been used by some members. (Also, as discussed in the comparative analysis section, the Canadian Institute for the Administration of Justice has designed a tailored two-day decision-writing course for members of the Social Security Tribunal.) In the case of ID members, the large majority of their work pertains to detention hearings and thus decisions are verbal and are made on the spot, i.e., they are not regularly required to produce written decisions.

      Some interviewees noted that the IRB does not provide members with sufficient training in conducting hearings (specifically, managing a hearing room) and writing decisions.

   4. Ongoing Training Related to Managing Caseloads and Workflows

      Training related to managing caseloads and workflows is done on an ad hoc basis, based on the needs of individual members. These needs are generally identified either by the member themselves or by their managers as part of performance reviews.

      The work of the ID is different than that of the other divisions: there are no backlogs because members clear their dockets daily. The work of the ID is a check/balance against the power of the CBSA whose border services officers have the authority to arrest and detain individuals without charge. The ID must conduct a review within 48 hours of the person being arrested and detained in order to determine whether they should be maintained in detention or released. The ID must make a decision at the hearing and decisions are oral.

      It was noted by interviewees that IRB lacks trainers with experience in the area of managing caseloads and workflows.

   5. Survey findings: satisfaction with the types of training provided^{Footnote 11}

      Figure 4 shows the extent to which the surveyed members undertook different types of training in 2016 and 2017. Almost 50% of the responding members undertook new member training in the last two years. Not surprisingly, the bulk of the participants in this training (92%) were members appointed to their current positions in 2016 or 2017. The most commonly undertaken ongoing training (that is, training subsequent to new member training) related to the substantive law and knowledge required to inform decision-making (by 87% of respondents), followed by training related to the conduct of hearings and making decisions (50%), and managing caseloads and workflows (21%). Incidence rates for each division were very similar to the overall breakdown across all divisions except for the ID where only one of the 13 participants (8%) reported participating in new member training and training on managing caseloads.

      Figure 4: Types of training undertaken in 2016 and 2017

      

      (Base: All respondents (n=113)
      

      • Text format

        This is a bar chart describing the types of training undertaken in 2016 and 2017 represented by 6 bars.

        The first bar is entitled ‘New Member Training’ and describes the percentage of respondents who received new member training. 46% of respondents received new member training.

        The second bar is entitled ‘Training-substantive law and knowledge for decision making’ and describes the percentage of respondents who received training in substantive law and knowledge for decision making. 87% of respondents received the aforementioned training.

        The third bar is entitled ‘Training- conducting hearings and making decisions’ and describes the percentage of respondents who received training in conducting hearings and making decisions. 50% of respondents received the aforementioned training.

        The fourth bar is entitled ‘Training-managing caseloads and workflows’ and describes the percentage of respondents who received training in managing caseloads and workflows. 21% of respondents received the aforementioned training.

        The fifth bar is entitled ‘Other’ and describes the percentage of respondents who received training in other areas. 10% of respondents received the aforementioned training.

        The sixth bar is entitled ‘None’ and describes the percentage of respondents who received no training. 2% of respondents received no training.

      Survey respondents were also asked to:

      • Rate their satisfaction with the different types of training provided to members.
      • Rate the extent to which these different types of training enabled them to make well-reasoned and timely decisions, efficiently manage their case loads and workflows, and immediately apply training content to member tasks.
      • Suggest ways of improving the IRB’s training programs.

      Participants’ ratings of their overall satisfaction with the different types of training provided are shown in Figure 5. These results show that majorities of those who participated in each type of training were either completely or mostly satisfied with the training they received, particularly for new member training and training covering the substantive law and knowledge required to inform decision-making. However, the relatively high levels of respondents indicating they were neither satisfied nor dissatisfied with training in conducting hearings and making decisions (18%), and training in managing caseloads and workflows (29%) suggests that these types of training are less effective for sizeable minorities of members. Respondents from the RPD were more likely to be satisfied with their training on conducting hearings and making decisions (81% versus 73% for all respondents) but slightly more likely to be dissatisfied with their training on managing caseloads and workflows (13% versus 8%).

      Figure 5: Satisfaction with the different types of training

      

      (Base: Respondents who participated in each type of training; new member – 52, substantive law – 98, conducting hearings – 56, managing caseloads – 24, other – 11.)
      

      • Text format

        This is a bar chart describing satisfaction levels with different types of training. It includes five bars.

        The first bar is entitled ‘New Member’ and describes the percentage of participants in new member training who were satisfied or unsatisfied with their training. 92% were completely or mostly satisfied with their training, 4% were neither satisfied nor dissatisfied and 4% were completely/mostly dissatisfied.

        The second bar is entitled ‘Substantive law and knowledge for decision-making’ and describes the percentage of participants who were satisfied with their training in substantive law and decision-making. 81% were completely or mostly satisfied with their training, 12% were neither satisfied nor dissatisfied and 7% were completely/mostly dissatisfied.

        The third bar is entitled ‘Conducting hearings and making decisions’ and describes the percentage of participants who were satisfied with their training in conducting hearings and making decisions. 73% were completely or mostly satisfied with their training, 18% were neither satisfied nor dissatisfied and 9% were completely/mostly dissatisfied.

        The fourth bar is entitled ‘Managing caseloads and workflows’ and describes the percentage of participants who were satisfied with their training in managing caseloads and workflows. 63% were completely or mostly satisfied with their training, 29% were neither satisfied nor dissatisfied and 8% were completely/mostly dissatisfied.

        The fifth bar is entitled ‘Other’ and describes the percentage of participants who were satisfied with their training in other areas. 73% were completely or mostly satisfied with their training, 9% were neither satisfied nor dissatisfied and 18% were completely/mostly dissatisfied.

      Respondents who received each of the various types of training provided by the IRB were also asked to rate the extent to which each type of training equipped them to make well-reasoned and timely decisions, to manage their case load and workflow efficiently and effectively, and provided knowledge and guidance that was immediately applicable in their work. The ratings of the different types of training against each of these outcomes are summarized in Figure 6.

      Figure 6: Utility of different types of training

      

      (Base: Participants in each type of training; new member: 52, substantive law: 98, conducting hearings: 56, managing caseloads: 24.)

      • Text format

        These are three bar charts describing the utility of different types of training.

        The first bar chart includes 4 bars and assesses whether training equipped respondents to make well-reasoned and timely decisions in his or her role as a member.

        The first bar is entitled ‘New Member Training’ and describes the percentage of participants in new member training that believe the training equipped them to make well-reasoned and timely decisions in his or her role as a member. 77% completely or mostly agree, 8% neither agree nor disagree, 13% completely or mostly disagree, and 2% responded that it was not applicable to him or her.

        The second bar is entitled ‘Training - Substantive law and knowledge’ and describes the percentage of participants in substantive law and knowledge training that believe the training equipped them to make well-reasoned and timely decisions in his or her role as a member. 62% completely or mostly agree, 21% neither agree nor disagree, 13% completely or mostly disagree, and 4% responded that it was not applicable to him or her.

        The third bar is entitled ‘Training - Conducting hearings and making decisions’ and describes the percentage of participants in conducting hearings and making decisions training that believe the training equipped them to make well-reasoned and timely decisions in his or her role as a member. 66% completely or mostly agree, 18% neither agree nor disagree, and 16% completely or mostly disagree.

        The fourth bar is entitled ‘Training – Managing caseloads and workflow’ and describes the percentage of participants in managing caseloads and workflow training that believe the training equipped them to make well-reasoned and timely decisions in his or her role as a member. 79% completely or mostly agree, 13% neither agree nor disagree, and 8% completely or mostly disagree.

        The second bar chart includes 4 bars and assesses whether training enabled respondents to manage his or her case load and workflow efficiently and effectively.

        The first bar is entitled ‘New Member Training’ and describes the percentage of participants in new member training that believe the training enabled them to manage his or her case load and workflow efficiently and effectively. 38% completely or mostly agree, 23% neither agree nor disagree, 37% completely or mostly disagree, and 2% responded that it was not applicable to him or her.

        The second bar is entitled ‘Training - Substantive law and knowledge’ and describes the percentage of participants in substantive law and knowledge training that believe the training enabled them to manage his or her case load and workflow efficiently and effectively. 36% completely or mostly agree, 29% neither agree nor disagree, 26% completely or mostly disagree, and 9% responded that it was not applicable to him or her.

        The third bar is entitled ‘Training - Conducting hearings and making decisions’ and describes the percentage of participants in in conducting hearings and making decisions training that believe that the training enabled them to manage his or her case load and workflow efficiently and effectively. 46% completely or mostly agree, 29% neither agree nor disagree, and 25% completely or mostly disagree.

        The fourth bar is entitled ‘Training – Managing caseloads and workflow’ and describes the percentage of participants in managing caseloads and workflow training that believe that the training enabled them to manage his or her case load and workflow efficiently and effectively. 58% completely or mostly agree, 29% neither agree nor disagree, and 13% completely or mostly disagree.

        The third bar chart includes 4 bars and assesses whether training provided respondents with knowledge and guidance that could be immediately applied to their work as a member.

        The first bar is entitled ‘New Member Training’ and describes the percentage of participants in new member training that believe the training provided them with knowledge and guidance that could be immediately applied to their work. 90% completely or mostly agree, 4% neither agree nor disagree, 4% completely or mostly disagree, and 2% responded that it was not applicable to him or her.

        The second bar is entitled ‘Training - Substantive law and knowledge’ and describes the percentage of participants in substantive law and knowledge training that believe that the training provided them with knowledge and guidance that could be immediately applied to their work. 83% completely or mostly agree, 10% neither agree nor disagree, 6% completely or mostly disagree, and 1% responded that it was not applicable to him or her.

        The third bar is entitled ‘Training - Conducting hearings and making decisions’ and describes the percentage of participants in in conducting hearings and making decisions training that believe that the training provided them with knowledge and guidance that could be immediately applied to their work. 75% completely or mostly agree, 16% neither agree nor disagree, and 9% completely or mostly disagree.

        The fourth bar is entitled ‘Training – Managing caseloads and workflow’ and describes the percentage of participants in managing caseloads and workflow training that believe the training provided them with knowledge and guidance that could be immediately applied to their work. 75% completely or mostly agree, 17% neither agree nor disagree, and 8% completely or mostly disagree.

      The main findings regarding the utility of the various categories of training in supporting the three outcomes are as follows:

      • Make well-reasoned and timely decisions. Majorities of participants in each of the four types of training provided to members (between 62% and 79%) believe that the training received does equip them to make well-reasoned and timely decisions. In this regard, new member training and training in managing caseloads and workflows was perceived to be more likely to aid this outcome. Small proportions (between 8% and 16%, depending on the type of training) did not agree that the training provided was effective in helping members to make well-reasoned and timely decisions.
      • Efficiently manage their caseloads and workflows. All four types of training were less likely to be perceived to enable members to manage caseloads and workflows efficiently and effectively. This was particularly true for new member training, training on substantive law, and training on conducting hearings and making decisions, where minorities of the respondents (38%, 36% and 46%, respectively) agreed their training contributed to this outcome. Only a small majority of those who participated in training in managing caseloads and workflows (58%) agreed that this training was helpful. RPD respondents had a higher incidence (45% versus 37% for all respondents) of disagreeing that their new member training enabled them to manage their case loads and workflows efficiently.
      • Immediately apply training content to member tasks. Participants in all four types of training were most likely to agree that the training they received provided knowledge and guidance that could be applied immediately, particularly for new member training (90%) and training on substantive law (83%). RPD respondents were more likely to be satisfied with the extent to which their training on conducting hearings and making decisions was immediately applicable to their work (81% versus 75% for all respondents).

      Respondents who perceived that the different types of training they received did not equip them to effectively or efficiently make decisions, manage their caseloads or immediately apply the knowledge gained were also asked how the different types of training provided could be improved or strengthened. While the numbers of respondents providing such suggestions was relatively low a number of common themes were apparent in their comments. These themes were as follows:

      • New member training:
      • More time on approaches to and structures for decision-making and reasons writing.
      • More time on preparing for and managing hearings, focusing questioning, and managing time and workflow.
      • More time on mock/practice hearings.
      • More time, or resources, on the legal basis and reasoning for decision-making, for example, provision of a compendium of up-to-date case law and case synopses on commonly occurring issues.
      • Members to lead/present on reasons writing, not Legal Services.
      • Systematic mentoring of new members by successful/effective members.
      • Follow-up meetings between new members and trainers, Legal Services and management at regular intervals post-training to review issues encountered and areas where additional help is needed, for example, once a month for six months or after six months and one year.
      • Training covering substantive law and knowledge required for decision-making:
      • Focus on emerging issues, issues arising in recent hearings/cases, and IAD/RAD and Federal Court decisions that have broader implications for IRB cases.
      • Legal Services should consult with members to identify substantive issues of concern before choosing and preparing topics for presentation.
      • Copies of training materials should be distributed prior to presentation to permit advance reading and preparation.
      • Present and/or structure training materials wherever possible as tools that facilitate case analysis and decision-making, and templates for efficient decision writing.
      • Allow for more discussion among members during training sessions regarding the implications of jurisprudential updates for the conduct of hearings and decision-making.
      • Divisional leaders need to involve LPDD and Legal Services in the planning and design of ongoing training to current members.
      • Provide Legal Services members/presenters with training on how to design and deliver training.
      • Spend time on foundational and contextual issues, such as the common law on misrepresentation (in IAD) and the implications of different levels of seriousness, and CBSA and IRCC perspectives on topical issues.
      • Training on conducting hearings and making decisions:
      • Conduct workshops on topics that affect the conduct and time required for hearings, such as:
        Time management in the hearing room
        Dealing with difficult counsel
        Dealing with under-representation and incompetent counsel
        Sufficiency of reasons
        What’s needed to write an adequate (not perfect) decision.
      • Focus on addressing knowledge and information gaps identified by members.
      • Ensure that Legal Services consult with members on their needs, understand how hearings work and decisions are made in practice, and avoid taking a directive tone in training content.
      • Have more mock hearings, simulations and case studies, and provide opportunities for members who are new to refugee or immigration issues to observe more hearings.
      • Allow time for discussion and interaction during training sessions.
      • Hold regular national in-person meetings of members.
      • Use properly qualified, well-informed and objective trainers and presenters, and avoid presenters with advocacy agendas.
      • Provide a training module on hearings for new RAD members to equip them with tools and methods needed if they have to conduct a hearing.
      • Training on managing caseloads and workflows:
      • More interactive training methods and discussions rather than being “talked at”, and more realistic.
      • (As an adjunct) encourage members to share their concerns and issues with their coordinating members with a view to finding suitable solutions.
   6. Survey findings: too little, too much or just the right amount of training?

      Survey respondents were asked if they thought too little, too much or just the right amount of training was being provided, and if they thought the amount was too little or too much, what subject areas were being either under- or over-covered.

      Figure 7 shows the breakdown of responses to the question about the suitability of the amount of training provided. These results show that large majorities of responding members believe the amount of new member training and training on substantive law and knowledge for decision-making is “just right” (76% and 79%, respectively). In contrast, both training on conducting hearings and making decisions, and managing caseloads and workflows, were both considered to be under-provided by 42% and 66% of the respondents.

      Respondents in the Western Region were noticeably likely to indicate they were getting too little new member training (31% vs. 14% overall), substantive law training (18% vs. 9%) and hearing conduct training (50% vs. 42%) compared to responding members in the Central and Eastern Regions. However, with regard to training on managing caseloads and workflows, members in the Central Region were most likely to believe they received too little (76% vs. 66% overall), Western members that they were just right (41% vs. 26%), and Eastern Region members received too much (23% vs 8%). RPD member respondents differed slightly in their assessments compared to the overall sample, being slightly more likely to indicate they received “too much” new member training (15% versus 10%), “too little” substantive law and knowledge training (13% versus 9%), and “too much” training on conducting hearings and making decisions (12% versus 7%).

      Figure 7: Too little, too much or just the right amount of training

      

      (Base: All respondents excluding those who selected ‘Not applicable; new member training: 88, substantive law: 107, conducting hearings: 102; managing caseloads: 103.)

      • Text format

        This is a bar chart that describes whether members believe they received too little, too much, or just the right amount of training in four training areas.

        The first bar is entitled ‘New Member Training’ and describes the percentage of respondents who believe that new member training was too little, just right, or too much. 14% indicated that they received too little training, 76% indicated that they received just the right amount of training, and 10% indicated that they received too much training.

        The second bar is entitled ‘Training - Substantive law and knowledge’ and describes the percentage of respondents who believe that substantive law and knowledge training was too little, just right, or too much. 9% indicated that they received too little training, 79% indicated that they received just the right amount of training, and 11% indicated that they received too much training.

        The third bar is entitled ‘Training - Conducting hearings and making decisions’ and describes the percentage of respondents who believe that conducting hearings and making decisions training was too little, just right, or too much. 42% indicated that they received too little training, 51% indicated that they received just the right amount of training, and 7% indicated that they received too much training.

        The fourth bar is entitled ‘Training – Managing caseloads and workflow’ and describes the percentage of respondents who believe that managing caseloads and workflow training was too little, just right, or too much. 66% indicated that they received too little training, 26% indicated that they received just the right amount of training, and 8% indicated that they received too much training.

      Subject areas where respondents indicated they received too little or too much training in the IRB’s training programs are summarized in Table 3. The preponderance of topics on the ‘too little’ list referring to aspects of hearing conduct and caseload management reflects the significance of these two types of training in the assessments of training sufficiency. The shortness of the list of ‘too much’ training areas reflects the small number of respondents who believe there are areas in which too much training is provided.

      Table 3: Areas of too little, and too much, training identified by responding members

      Areas With Too Little Training

      Areas With Too Much Training

      Conducting hearings: Decision-making/reasons writing/rendering bench/oral decisions Managing hearings Assessing the credibility of evidence/reasons Questioning skills/techniques Dealing with difficult/inexperienced counsel Dealing with difficult/troublesome claimants/appellants Caseload/Workflow Management: Caseload/workload/workflow Management Time management/improving efficiency of work Substantive Law and Knowledge: Country-specific factors and considerations Adjudicating de novo claims Specific substantive law and knowledge topics

      Conducting hearings: Conducting a hearing/reviewing a file Decision writing/workflow management/ time management (for experienced members) Substantive Law and Knowledge: Legal updates that have no direct/new consequences for members’ work or that repeat coverage of issues Coverage of infrequently occurring areas of practice New Member training: Split new member training to cover/reinforce elements after some time on the job Other: Length of training is too long

      (Base: Survey respondents who selected ‘too little’ or ‘too much’ training, and provided suggestions regarding the relevant subject areas)

2. Modes of Training Delivery

   Summary: The IRB conducts most training in-person and this, based on the key informant interviews and the survey of members, is the mode most preferred by members. However, the IRB is increasingly experimenting with other modes such as videoconferencing and webinars (using WebEx software), with varying degrees of success. A formalized process of mentoring is also being used increasingly as part of new member training and this is viewed favorably by members.

   With regard to specific aspects of the delivery formats used by the IRB, the survey found that members are most satisfied with the provision of training in both official languages, the quality of instructors, opportunities to question and discuss content with presenters, and the quality of supporting materials and handouts. Members are less satisfied with opportunities to discuss content with other training participants, to provide feedback on the appropriateness and quality of the training received, and to practise the application of content during training sessions. These are all key success factors for effective adult learning programs.

   This section presents evaluation findings related to the effectiveness of the delivery formats used for member training and professional development.

   1. Interviewees' Perspectives on the Delivery Formats Used

      The IRB conducts most of its member training in-person. There was consensus among interviewees that this was both the most appropriate mode as well as the mode most preferred by members in all divisions. Interviewees explained that members appreciate the opportunity to attend conferences and other in-person training. The work members do tends to be very solitary and being able to talk to colleagues and exchange experiences is valued. A few interviewees noted that the in-person group training needs to be dynamic and engaging in order to be useful – members need to be engaged and encouraged to discuss material among themselves.

      Mentoring is seen by interviewees as a key component of new member training – “mentors are absolutely necessary,” and “every new member should have a mentor”. A few interviewees noted that new members who are mentored reach performance goals more quickly than those who are not assigned a mentor. A benefit of mentoring is that it enables the division to reinforce the use of some of the tools presented in the training materials used at the new member training course (such as decision trees). It was noted that when new members ask for mentoring, they feel they need someone to read their decisions or talk to about them. A challenge raised with respect to mentoring is the lack of a sufficient numbers of experienced members who can take on this role.

      Two divisions, RAD and IAD, have established mentoring processes as part of their new member training. In RAD, there is a mentoring team for new members consisting of the ADC, a representative from Legal Services and a mentor who spends between four and eight weeks with the new member. Typically, the mentor reviews each file in parallel with the member, then spends time discussing the issues with the member. Sometimes there are discussions with Legal Services. The mentor then helps the members understand the commentary provided by Legal Services (which the member may or may not accept) before finalizing the decision. In IAD, new members are paired with an experienced member who observes the new member conducting hearings and then reviews the proposed reasons and provides advice. The reasons are also reviewed by Legal Services.

      Although all IRB divisions have used videoconferencing and webinars, interviewees generally agreed that the IRB struggles with using these newer communications technologies. Technological challenges include lack of a wireless connection to the Internet in some regions along with bandwidth limitations, which limit the number of people that can participate in a webinar. Despite these challenges, interviewees also noted the benefits; for example, videoconferencing is a cost-effective way to bring together people from different regions or offices.

      According to interviewees, the use of online self-study at the IRB is increasing. For example, RPD, RAD and IAD are using more pre-course self-study for new member training. However, interviewees also noted that, in general, members do not like this mode of training.

   2. Survey Findings: Satisfaction with Different Delivery Formats

      This section presents the results of the member survey regarding satisfaction with the main formats used to deliver training and professional development, satisfaction with specific aspects of the delivery formats, and views regarding the relative effectiveness of each format.

      Figure 8 summarizes the breakdown of respondents’ satisfaction ratings of the four main formats used by the IRB: in-person group seminars and presentations, one-on-one interactions (such as, for example, mentoring), online webinars and interactive videoconferences, and online self-study and research. The percentage distributions of responses shown in the table are after respondents who indicated that the delivery formats were ‘not applicable’ – presumably because they had not participated in training using the particular delivery format in question.

      These results show that large majorities of members who have participated in in-person group training (80%) and one-on-one training and/or mentoring (68%) were either completely or mostly satisfied with these formats. In contrast, minorities of the responding members were completely or mostly satisfied with online webinars and interactive videoconferences (36%), and online study and research (40%). Levels of dissatisfaction with these two formats were relatively high, at 38% and 23% respectively. RPD respondents were more likely to feel to be dissatisfied with the one-on-one training they received (19% versus 14% for the total sample), less likely to be satisfied with on-line webinar presentations and videoconferences (27% versus 36%), and more likely to be dissatisfied with training via on-line self-study and research (29% versus 23%).

      Figure 8: Satisfaction with training delivery formats

      

      (Base: All respondents excluding those who indicated a delivery format was ‘Not Applicable’; (a)=111, (b)=81, (c)=84, (d)=98.)

      • Text format

        This is a bar chart that describes respondents’ satisfaction with training delivery formats. It includes 4 bars.

        The first bar is entitled ‘In-person group seminars and presentations’ that describes respondent satisfaction with in-person group seminars and presentations. 80% indicated they were completely or mostly satisfied, 13% indicated they were neither satisfied nor dissatisfied, and 7% said they were completely or mostly dissatisfied.

        The second bar is entitled ‘One-on-one (e.g. mentoring)’ that describes respondent satisfaction with one-on-one training. 68% indicated they were completely or mostly satisfied, 19% indicated they were neither satisfied nor dissatisfied, and 14% said they were completely or mostly dissatisfied.

        The third bar is entitled ‘Online webinar presentations and interactive videoconferences’ that describes respondent satisfaction with online webinar presentations and interactive videoconferences. 36% indicated they were completely or mostly satisfied, 26% indicated they were neither satisfied nor dissatisfied, and 38% said they were completely or mostly dissatisfied.

        The fourth bar is entitled ‘On-line self-study and research’ that describes respondent satisfaction with on-line self-study and research. 40% indicated they were completely or mostly satisfied, 37% indicated they were neither satisfied nor dissatisfied, and 23% said they were completely or mostly dissatisfied.

      The most commonly-mentioned aspects of the IRB’s training formats considered to be most effective, and least effective, are summarized in Table 4. These results are consistent with those shown in Figure 8, and also show that views among members are not consistent, for example, while many find videoconferences ineffective, a smaller proportion views this delivery format as being effective.

      Table 4: More, and less, effective aspects of IRB delivery formats

      Most Effective Delivery Aspects	# of Mentions	Least Effective Delivery Aspects	# of Mentions
      In-person group training sessions	57	Videoconferences - difficult to manage and enable discussions with large groups, and often suffer from technology problems	36
      One-on-one mentoring	32	Online webinars/courses	20
      Opportunities to exchange ideas/ experiences in group sessions/ Interactive training sessions	7	Online self-study	19
      Jury/juri meetings	6	In-person group sessions with limited opportunities for discussion/ interaction, members are lectured to, or presenters read their presentations.	12
      Case studies/ practical exercises and examples/ Mock hearings	6	Mentoring (including lack of opportunities to obtain mentoring)	5
      Presentations/classes by Legal Services/ Reviews of recent jurisprudence	4	Limited availability of training materials – paper copies, online copies	3
      National videoconferences/ Online webinars	4	Usefulness of Legal Services’ jurisprudential updates	2
      On-line self-study	3	Availability of French and English translation and/or bilingual trainers	2
      Presentations by subject matter experts	3	Poor training skills of trainers	2
      Other aspects	6	Lack of, or poorly structured, methods for monitoring the initial performance, and providing feedback to, new members	9

      (Base: Respondents who provided examples of most (n=104), and least (n=96), effective delivery formats.)

      Respondents were also asked to indicate their levels of satisfaction with various specific aspects of the delivery of training, which are summarized in Figure 9. Members are most likely to be completely or mostly satisfied with the provision of training in both official languages (88% of respondents), the quality of instructors and their command of the subject matter (81%), opportunities to ask questions and discuss content with presenters (78%), and the quality of supporting materials and handouts (78%). Ratings by RPD respondents were very similar to the patterns for the overall sample except for being more likely to be dissatisfied (21% versus 15%) or neither satisfied nor dissatisfied (25% versus 19%) with opportunities to discuss content with other members in training sessions, and more likely to be dissatisfied (20% versus 16%) with opportunities to provide feedback on the appropriateness and quality of the training received.

      Figure 9: Satisfaction with specific aspects of the delivery of training

      

      (Base: All respondents excluding those who selected ‘Not applicable; sub-sample sizes: 1.-94, 2.-108, 3.-110, 4.-109, 5.-108, 6.-104, 7.-107, 8.-104.)
      

      • Text format

        This is a bar chart that describes respondents’ satisfaction with specific aspects of the delivery of training. It includes 8 bars.

        The first bar is entitled ‘Provisions of training in the official language of choice’ and describes satisfaction with training in respondents’ official language. 88% were completely or mostly satisfied, 9% were neither satisfied nor dissatisfied, and 3% were completely or mostly dissatisfied.

        The second bar is entitled ‘Quality of instructors and their command of the subject matter’ and describes satisfaction with instructors. 81% were completely or mostly satisfied, 11% were neither satisfied nor dissatisfied, and 8% were completely or mostly dissatisfied.

        The third bar is entitled ‘Opportunities to ask questions and discuss content with presenters’ and describes satisfaction with opportunities to engage presenters. 78% were completely or mostly satisfied, 15% were neither satisfied nor dissatisfied, and 6% were completely or mostly dissatisfied.

        The fourth bar is entitled ‘Quality of supporting handouts and reference materials’ and describes satisfaction with supporting handouts. 78% were completely or mostly satisfied, 15% were neither satisfied nor dissatisfied, and 7% were completely or mostly dissatisfied.

        The fifth bar is entitled ‘Opportunities to discuss content with other members in attendance’ and describes satisfaction with opportunities to engage other members. 67% were completely or mostly satisfied, 19% were neither satisfied nor dissatisfied, and 15% were completely or mostly dissatisfied.

        The sixth bar is entitled ‘Opportunities to provide feedback on the appropriateness and quality of the training’ and describes satisfaction with opportunities to provide feedback. 61% were completely or mostly satisfied, 23% were neither satisfied nor dissatisfied, and 16% were completely or mostly dissatisfied.

        The seventh bar is entitled ‘Opportunities to practice the application of content, e.g., via group exercises, case studies’ and describes satisfaction with opportunities to apply learning. 57% were completely or mostly satisfied, 20% were neither satisfied nor dissatisfied, and 23% were completely or mostly dissatisfied.

        The eighth bar is entitled ‘Suitability and reliability of technologies for content delivery’ and describes satisfaction with technological delivery of content. 48% were completely or mostly satisfied, 29% were neither satisfied nor dissatisfied, and 23% were completely or mostly dissatisfied.

      However, in four other areas the levels of satisfaction/dissatisfaction are more mixed, of which three relate to substantive elements of adult learning approaches. These three areas relate to opportunities to discuss content with other members in attendance (67% satisfied), opportunities to provide feedback on the training received (61%), and opportunities to practice the application of content during the training (57%). Just slightly less than half of the respondents (48%) were satisfied with the final aspect, suitability and reliability of technologies used for delivery purposes, as was also highlighted in Table 4, above.

Adequacy of Resources

This section addresses the adequacy of resources available for member training and the efficiency of the organizational structures for the planning, design and delivery of member training. The IRB does not currently have a system for tracking the human and financial resources devoted to member training and professional development and thus the findings in this section are based largely on qualitative information collected through interviews with IRB management.

1. Adequacy of Resources

   Summary: Overall, the evidence suggests that resources for member training and professional development are adequate. However, existing resources could be used more effectively and efficiently through better coordination and collaboration across divisions and within divisions across regions. There is evidence of a need for increased administrative support for Legal Services when training is being delivered as well as increased funding for translation of existing training materials so that they can be better shared across regions.

   Views were divided on whether resources for member training are sufficient. Those who believe that resources are sufficient noted that requests for training have always been met. Several interviewees within Legal Services emphasized that the needs of members are of utmost importance and therefore training requests are never denied. Sometimes new GIC members are appointed with little notice (just a few weeks) and Legal Services always ensures that the newly appointed members receive their new member training upon arrival.

   The main concerns with resources were related to insufficient administrative support for training and inadequate budgets for translation of training materials. Specifically the Eastern Region would like to implement the new modules developed for new member training in RPD but as of early 2018 these modules have not been translated due to a lack of financial resources.

   Comments by a number of interviewees indicate that a greater level of involvement by members in the planning and delivery of training does have an opportunity cost in that these members then have less time available for hearings and/or decision-making. These interviewees supported the involvement of members in the provision of training (“members teaching members”) but suggested that expectations regarding the volume of decision-making by these members should be reduced to recognize the time required for their contributions. While not stated in the interviews, there appears to be an implicit expectation that the involvement of experienced members in the provision of new member training will lead to faster rates of ramp-up by new members and more efficacious decision-making.

   Although LPDD has been called upon to provide training support in IAD and sometimes provides administrative support to other divisions in delivering training, LPDD managers indicated that the directorate no longer has resources available to support member training.

   According to Legal Services in the Central Region, a one-month time capture study completed in late 2017 found that 23% of the time of Legal Services personnel was devoted to training, which was higher than expected. Furthermore this estimate was conservative as the study was undertaken at a time when no new member training was being designed or delivered. Several interviewees commented that considering the relatively high salary ranges of lawyers in Toronto^{Footnote 12}, the IRB is devoting a significant amount of human and financial resources to member training.

2. Efficiency of Organizational Structures for Member Training

   Summary: There is evidence that the efficiency of the organizational structures for delivering member training and professional development could be improved. There is a lack of coordination within divisions and nationally. In addition, there is no central repository of training materials which limits the sharing of materials across divisions and regions.

   There is broad agreement among interviewees from all divisions that the organizational structure for the provision of member training at the IRB is not efficient. According to interviewees, in the past, there were national and regional professional development committees but now each division and region undertakes member training largely on its own. Some noted that in the past there was a national professional development committee. Having such a committee would improve coordination and allow for the sharing of training materials and best practices.

   Interviewees within Legal Services noted that a central repository for training materials does not exist (either within each region or nationally), which means time is wasted searching for the latest training materials. This may be resulting in duplication with divisions and regions developing similar training in isolation. In addition to the potential for duplication, some noted the issue of a lack of consistency in training across regions of each division.

Alignment with IRB Mandate and Priorities

Summary: There was agreement among interviewees that the IRB’s member training, in all divisions, aligns with its mandate. Some noted that the mandate is a key driver for the changes to the training approaches being implemented in RPD, RAD and IAD.

With respect to the IRB’s two strategic priorities, one, “limiting the growth of the refugee determination backlogs” primarily relates to the work of the RPD, and to a lesser extent, the RAD, and the second, “eliminating the backlog of immigration appeals”, is only applicable to the work of the IAD. Interviewees from the divisions who commented on the degree of alignment noted that training includes attention to controllable factors that contribute to the size of the backlog, such as, minimizing postponements and preparing decisions in a timely manner. However, while the survey of members did not ask about the degree of alignment against these priorities directly, responses to the question about the sufficiency of training (presented earlier, in the section of effectiveness of training) suggest that training could be more closely aligned. In particular, significant proportions of the surveyed members indicated that they felt they received too little training in managing caseloads and workflows (66%), and conducting hearings and making decisions (42%).

Best Practices and Alternatives

Summary: Interviewees identified several improvement opportunities including: 1) consider compressing the length of new member training; improve the national coordination of training; clarify roles and responsibilities regarding the provision of member training; and, develop training capacity, particularly by ensuring each division has access to adult learning specialists.

This section identifies best practices and alternatives that could be implemented by the IRB to improve the efficiency and/or effectiveness of the planning, design or delivery of member training. Findings in this section are based on key informant interviews with IRB managers and members as well as a comparative analysis of how training is delivered at the NJI and SST.

1. Suggested Alternative Approaches
   1. Improve National Coordination of Training

      There was agreement among interviewees that increased national coordination and collaboration would increase both the effectiveness and efficiency of planning, development and delivery of member training. Although there is acceptance that there are differences in training needs across divisions and regions, there is also a sense that there are some needs and skill sets common to all divisions and regions. Common training needs include things such as dealing with difficult counsel, dealing with mental health issues in claimants, decision-writing, and time and caseload management.

      In addition, the lack of a central repository for the sharing of training materials means that each division and region is developing its own training independently – this is seen as inefficient and doesn’t allow for the sharing of best practices.

      The lack of national coordination and a central repository of training materials suggest a lack of consistency in training across regions within divisions. For example, interviewees noted that the new adult learning-based approaches to new member training have not yet been applied across all regions of RPD, RAD and IAD. The lack of consistency in training for new members implies a potential for a lack of consistency in decision-making across members in different regions. There is agreement that a more consistent approach to training needs to be implemented.

   2. Clarify Roles and Responsibilities

      Some interviewees suggested that roles and responsibilities for member training be clarified with a stronger presence within senior IRB leadership for member training. For example, prior to 2012 there was a national level group responsible for training at the IRB that was led by a senior person at the level of a DC with significant experience in training in tribunals.

      Interviewees also suggested that the nature of the roles played by LPDD and Legal Services in member training be clarified, to reinforce that these organizations are service providers that take their direction from the members (or more specifically, the DCs and ADCs). One such respondent summarized this positioning as, “LPDD and Legal Services are not the owners of the training”.

   3. Develop Training Capacity in IRB

      Interviewees indicated that the IRB needs members, learning designers and Legal Services staff who are skilled in learning design for adult learners. The need for Legal Services to have improved skills in developing and delivering training to adult learners was noted by interviewees. Legal Services staff met in February 2018 for a one-day workshop on adult training methods based on the ERGA method developed by St. Francis Xavier University.

2. Approaches of Other Organizations
   1. Social Security Tribunal

      The Social Security Tribunal of Canada (SST) is an independent administrative tribunal that makes quasi-judicial decisions on matters relating to the Canada Pension Plan, the Old Age Security Act and the Employment Insurance Act.

      The Tribunal’s mandate is to offer fair, transparent, credible and impartial appeal processes that are efficient and effective.

      The SST consists of a Chairperson, three Vice-Chairpersons and members, all of whom are appointed by Governor-in-Council. The Tribunal comprises a General Division and an Appeal Division. The General Division – Employment Insurance Section hears appeals and makes decisions on appeals of reconsideration decisions that are made by the Canada Employment Insurance Commission. The General Division – Income Security Section hears appeals and makes decisions on appeals of reconsideration decisions that are made by the Minister of Employment and Social Development Canada regarding the Canada Pension Plan and Old Age Security Act. The Appeal Division decides matters on appeal from the General Division. Each decision is rendered by a single member.

      The SST has a total of 216 employees, including 85 members who are all GICs, with a budget of $23.8 million for 2017-18.

      There is a major structural difference between the IRB and SST that affects the provision of member training services. Support services to the SST are provided by a separate organization, the Administrative Tribunals Support Service of Canada (ATSSC). The ATSSC provides support services and facilities to a total of eleven federal administrative tribunals. Each tribunal is supported by its own secretariat – in this case, the Secretariat to the Social Security Tribunal. The ATSSC provides specialized services tailored to and required by each tribunal (e.g., registry, research, legal) as well as common shared internal services (HR, finance, IM/IT, etc.).

      Thus Legal Services within the ATSSC is responsible for member training. The General Counsel, Legal Services works closely with the SST Chair and Vice-Chairs to ensure that the members receive the required training. In contrast to the IRB, members are not involved in the delivery of training. The ATSSC IM/IT specialists provide training on the case management system.

      The SST recently underwent a comprehensive external review by KPMG in response to a directive of the Minister of Families, Children and Social Development. The objective was to improve the SST’s processes to ensure that it meets the needs and expectations of Canadians, and to assess its fairness and transparency.^{Footnote 13} The KPMG report notes that in contrast to other tribunals, the SST does not require the ATSSC to assist members with most aspects of the cases or decisions. In other tribunals (including the IRB), it is common for staff to provide advisory services to members, including research tasks, decision review for consistency, etc.

      ATSSC Legal Services delivers a two-week new member training course, which is held in Ottawa. While Legal Services indicated that there is probably too much material to cover in two weeks, resource pressures prohibit increasing the length of the course. In contrast to the IRB, all new members from across Canada come to Ottawa for this course. In 2017-18, approximately 25 session weeks were held. The large number of sessions reflects the fact that new members tend to be appointed in small batches (as is the case at the IRB). Legal Services intends to shorten the length of the new member training course, as the SST believes the amount of resources devoted to this activity is too high. An option being considered is for members to do one week of online interactive training from home followed by assembling in Ottawa for a one-week course.

      One of the training priorities is to improve decision writing. The KPMG review also recommended that decisions needed to be simplified in order to better serve client needs. Recognizing that decision writing is not a core competency of lawyers, Legal Services engaged the Canadian Institute for the Administration of Justice (CIAJ) to develop a two-day course on writing decisions. The SST reports that this outside course has been well received by members.

      Beyond the new member training, all members come back to Ottawa once a year for the two-day session on decision writing (offered by the CIAJ) and one day on jurisprudential updates.

      In contrast to the IRB, the SST has made limited use of webinars and videoconferencing, but would like to do online training in the future.

      In summary, the main findings of the SST comparative analysis are as follows:

      • A separate organization, the ATSSC, provides support services to the SST. Legal Services within the ATSSC provides training services to SST members, working closely with the SST’s Chairs and Vice-Chairs. Members are not involved in the delivery of training.
      • The SST’s new member training course is two weeks in length and is held in Ottawa at frequent intervals, due to the fact that new members, who are GICs, are appointed in small batches.
      • The SST intends to shorten the new member training course, by having members do one week of at-home online preparation, followed by one week in Ottawa. This is being done to reduce the amount of resources devoted to this course.
      • A priority has been to improve decision writing. Legal Services recognized that it lacked sufficient expertise to provide this type of training. An outside organization, the Canadian Institute for Administration of Justice, developed a tailored two-day course for the SST. All members are obliged to come to Ottawa to take this course, which is supplemented by one day of jurisprudential updates.
      • The main suggested best practice is to employ outside specialists to provide specific types of training (such as decision writing) for which the required expertise is not available in-house.
   2. National Judicial Institute

      The National Judicial Institute (NJI) is an independent not-for-profit organization with a mandate to coordinate and deliver education to members of the judiciary across Canada. The funding structure for NJI, established under the Judges Act, means that the primary clientele for the NJI’s services is federally-appointed judges working in the superior courts across Canada.

      In doing so, the NJI characterizes its approach to judicial education as being “judge-led, judging focused, experiential learning”. Taking a judge-led approach means that judges play leading roles in the design and delivery of judicial education, and the education provided is immediately applicable and respects judicial independence. The experiential learning approach means that lectures are used in combination with a mix of interactive methods, including panel discussions, participant debates, simulations based on real-life experience, and practice sessions.^{Footnote 14}

      In meeting its mandate, the NJI offers two major program streams – national programs containing courses that all federally-appointed judges are authorized by the Canadian Judicial Council to attend, and court-requested programs that cater to the training and education needs of judges working in particular provincial and territorial courts. Chief justices of the applicable courts approve the provision of these courses for their judges.

      The NJI employs approximately 50 people comprising lawyers (counsel) who liaise with their assigned courts to determine needs and work with course working groups to coordinate course planning and development, including providing legal input/ perspectives; events planners who work with the legal counsel and course working groups on the logistical arrangements; administrative support functions; and, an international group that provides international judicial education services funded by Global Affairs Canada. The NJI does not currently employ any experiential learning specialists but engages their services on an as-and-when required basis. Legal counsel and events planners represent approximately 50% of the staff.

      Formalized needs assessments are undertaken from time-to-time; for example, a defined needs assessment is currently underway, including a survey of the approximately 1,150 federally-appointed judges served by the NJI, as part of a strategic planning exercise being undertaken by the Canadian Judicial Council (CJC). Aside from these periodic exercises, the identification of needs for new educational content is ‘organic”, based on contacts between the courts and the NJI’s program managers and counsel as well as contacts between senior judges and members of the NJI’s Board of Governors, chaired by the Chief Justice of Canada. For example, the recent development of a course on terrorism trials came about as a result of a single call to the NJI seeking assistance with these types of complex and lengthy trails.

      NJI’s curriculum has three subject themes, with the various courses applicable to differing levels of judicial experience, described below. In practice, courses often combine aspects of all three thematic areas to improve the experiential nature of the training provided.

      • Content of judging – courses that address the substantive law or knowledge from other disciplines that judges need to be cognizant of in their roles.
      • Craft of judging – courses designed to increase judicial skill and the performance of tasks related to such areas as managing court and settlement processes (“judge craft”), judgement writing and communication (“court craft”), and development of supporting skills, such as research skills (“profession craft”).
      • Social context for judging - courses that address the environment and institutions within which judging occurs.

      Within this mix, attention to managing caseloads and workflows does not receive as much attention as substantive law, managing hearings and writing judgements but is taking on increasing importance in the light of more intensive workloads for judges and the Supreme Court’s ruling on reasonable time frames for cases to come to court. Current coursework relating to managing caseloads includes training on managing long trials and complex trials as well as court leadership and management for senior judges.

      Training for new judges is an important part of the services provided by the NJI. Introductory training for new federal judges is mandatory and is delivered in collaboration with the Canadian Institute for the Administration of Justice, while training for new provincial and territorial judges is delivered in collaboration with the Canadian Association of Provincial Court Judges and the Ontario Court of Justice. The “Seminar for New Federally Appointed Judges” is provided annually, in two six-day blocks and spans combinations of substantive law, hearing and trial management, and judgement writing. This seminar is designed to refresh new judges’ understanding of all areas of the law (criminal family, civil) and provide the foundation for their role. The NJI is also developing more specialized intensive courses covering specific areas of judicial practice for delivery in years one and two of the new judges’ careers, and considering, with the CJC, the feasibility of requiring new judges to have professional development plans to guide their continuing education during their initial five years on the bench.

      The primary delivery format for NJI courses is through in-person interactive seminars and fora. Some use has been made of online courses – mainly self-managed asynchronous courses conducted over periods of three to five weeks – with limited success or satisfaction on the part of participants. This approach was used to enable the participating judges to maintain their existing work schedules and to take the number of times zones in Canada into account. Other ways of using online delivery formats are being tried, for example, the NJI is currently developing a series of short (30 minute) videocasts on specific topics that include summaries of the subject matter and legal context, and provide links to supporting materials, which will be available for viewing by judges at any time. Electronic bench books are also maintained on various topics, and generally consist of short summaries of the law, checklists, access to relevant legislation and cases, as well as access to papers and other electronic resources. Copies of all materials used in training courses are also accessible to judges in the NJI’s electronic judicial library. The current needs analysis survey being conducted for the NJI and CJC includes a focus on the potential for further application of online methods.

      Best practices considered to be at the heart of the NJI’s success include:

      • Maintenance of close relationships with the senior leadership of the various courts across Canada by program managers and counsel to support their efforts to identify and respond to needs for judicial education, functioning as a highly responsive service provider.
      • Attention to needs to develop judicial skills and facilitate the understanding of the social context in which the judiciary, and judges, function. These judicial craft and social context aspects of judicial skills complement the attention to substantive law and knowledge that forms the third dimension of the NJI approach to judicial education. The NJI is a pioneer in the field of social context education for the judiciary and continues to develop and broaden the range of subjects covered in response to emerging needs.
      • A strong emphasis on judge-led education that integrates substantive content with experiential approaches to transferring knowledge and developing skills that provide time for participants to debate the issues and practice the application of new knowledge and skills. Input from subject matter specialists and advisers is also necessary, and supports the transfer and translation of knowledge.
      • Recycling of course materials and approaches wherever possible, given that many training and education needs are “evergreen”.
      • Provision of electronic access to all course materials for use on as a resource by all judges on an as-and-when-needed basis.

Conclusions and Recommendations

Conclusions

The conclusions of the evaluation on each of the issues and questions are summarized below.

The roles and responsibilities for the provision of member training need to be clarified and communicated: The identification of member training needs and the development and delivery of training services is mainly the responsibility of two groups: the four divisions (which define training needs and may be involved in the design and delivery of training sessions) and Legal Services (substantive law experts who also help to identify member needs and develop and deliver the training). The DCs and ADCs are viewed by Legal Services as the primary clients for member training. Aside from helping to re-design new member training within IAD, the LPDD has had limited involvement in member training in recent years. LPDD indicated that it no longer has the resources to support this activity. Most key informants indicated that the roles and responsibilities for member training are not clear; for example, if LPDD is no longer to be involved in the design and delivery of member training, then this should be communicated.

The appropriate organizational model for the provision of member training would reflect the following key elements to systematically address the knowledge and skills required to assess evidence and parties, conduct hearings and make decisions, and manage caseloads.

1. A systematic approach to the identification of member training needs that would be led by each division and formally shared across divisions;
2. A collaborative approach to the design and delivery of training.
3. A formal approach to the ongoing evaluation of member training, the results of which would feed back to update an annual member training plan. Each of these key elements is described further below.

Successful provision of these elements depends on the involvement of experienced members; subject-matter experts who provide perspectives on substantive law, best practice approaches to hearing management, decision writing and caseload management; and, adult learning advisers who assist with the design of courses and selection of delivery methods and formats.

A more systematic approach to the identification of member training needs is required: The identification of member training needs is a collaborative process involving the four divisions and Legal Services, and, as indicated above, the divisions take the lead. The IRB does not have a national, nor divisional, professional development committees that would identify the common training needs across divisions and coordinate the provision of training across the organization. In general terms, DCs appear to be satisfied with the current informal approaches to the identification of training needs.

Overall, two-thirds of members surveyed are satisfied with the needs identification process, indicating that there is room for improvement. Suggestions mostly related to systematically soliciting member input and establishing individual training plans as part of the member performance review process.

The approaches to the planning, design and delivery of member training vary by division and region: In general, the ADCs or DC of each division have the final say on the agenda for each training session, including new member training as well as ongoing professional development. There is agreement that the design of training sessions in all regions is done largely by Legal Services with assistance/input from members. Little training is planned or designed in collaboration with LPDD. Three divisions – RPD, IAD and RAD – have redesigned their new member training programs based on adult learning principles. As of the time of our research in early 2018 the ID had not decided whether to adopt the new approach. Ongoing training is delivered almost exclusively by Legal Services in all divisions and regions.

There is no formal process to monitor and evaluate the extent to which training is applied and desired performance outcomes achieved. Adult learning specialists could be contracted to design standardized feedback and evaluation tools for application by the divisions or to conduct periodic assessments of the effectiveness of the various types of training provided. These assessments would then inform the ongoing modifications and development of training programs and courses within each division.

DCs are generally satisfied with the provision of new member training but believe ongoing professional development can be made more effective and efficient. DCs are generally satisfied with the provision of new member training, and the approaches introduced in the RPD, RAD and IAD are perceived to be better equipping new members to understand and interpret the applicable substantive law. They are less satisfied with the provision of ongoing professional development, in terms of equipping members to manage hearings, make decisions and manage caseloads, and the coordination of planning and information sharing between regions. Most DCs also identified that bringing members together for national events can be very productive, in that a lot of learning is a function of both subject matter and related discussions between members.

Members are satisfied with the new member training and ongoing training covering substantive law and the knowledge required for decision making. Members are less satisfied with ongoing training related to conducting hearings and making decisions and managing caseloads and workflows. The new re-designed new member training programs in the three divisions based on adult learning principles are seen as an improvement to the previous approaches. Some interviewees were concerned with the increased time and resources required to train new members using the new approaches. An analysis of trends in new member productivity in the RPD Central Region indicates the new approach may be helping new members achieve full productivity in a shorter length of time, although it is too early to draw a definitive conclusion. The findings of the survey of members show that a vast majority of members were satisfied with the training they received, particularly for new member training and training covering the substantive law and knowledge required to inform decision-making. Ongoing training related to conducting hearings and making decisions and managing caseloads and workflows is not entirely meeting the needs of members. Similarly, significant proportions of members indicated that they were receiving insufficient amounts of these types of training.

Members prefer in-person training compared to other modes. The IRB conducts most training in-person and this is the mode most preferred by members. The IRB is increasingly experimenting with videoconferencing and webinars, with varying degrees of success. A formalized process of mentoring is also being used increasingly as part of new member training and this is viewed favorably by members.

The level of resources devoted to member training is adequate, although there is insufficient administrative support and resources for translation of materials. The evidence suggests that the level of resources devoted to member training and professional development is adequate. However, existing resources could be used more effectively and efficiently through better coordination and collaboration across divisions and, within divisions, across regions. Legal Services has a need for increased administrative support for training. Increased funding for translation is required.

Operational efficiency of member training could be improved: There is a lack of coordination within divisions and nationally. There is no central repository of training materials which limits the sharing of materials across divisions and regions. Addressing these issues would result in greater effectiveness with a reduced level of resources devoted to training overall.

A number of improvements were suggested: While new member training received high marks from members, some interviewees wondered if the amount of resources devoted to designing and delivering these courses could be reduced. One of the comparison organizations (Social Security Tribunal) provides its new member training program in two weeks. Interviewees generally agreed that the IRB needs to increase the national coordination and collaboration of member training, which also would help to ensure that common approaches are used to address such common needs as dealing with difficult counsel, dealing with mental health issues, decision-writing and time and caseload management. A central repository for all training materials is needed. The organizational model for member training, including the roles and responsibilities of Legal Service and LPDD, needs to be documented and communicated. Given the IRB is moving towards training based on adult-learning techniques, the capability of the organization (particularly within Legal Services) to provide this type of training needs to be increased.

Recommendations

Our evaluation of the IRB’s learning and professional development function found that, while majorities of the participating members were satisfied with the training and professional development they received, certain needs are not being well-met and opportunities exist to improve and better integrate training and professional development approaches. The most important of these needs are to:

• Ensure that training and professional development is built around members teaching members, supported by suitable subject matter and delivery expertise.
• Implement a systematic and integrated approach to new member training and ongoing professional development.
• Increase the provision of training on:
• Conducting hearings and decision writing.
• Managing caseloads and workflows.
• Incorporate adult learning approaches into the design and delivery of new member training and ongoing professional development.

However, in order to achieve an optimal combination of effectiveness and efficiency, responses to these needs should positioned within a broader strategic context for managing, planning, designing and delivering member adjudicative training. Our recommendations are thus grouped under the major parameters for this strategic context.

1. Strengthen the oversight and direction for member training

   The current approach to member training in the IRB is fragmented, with new member training largely planned and delivered independently of ongoing professional development, and ongoing professional development largely undertaken on a region-by-region basis within each division. A more integrated approach to direction setting for, and oversight of, member training should ensure more consistent development of members’ knowledge and skills, and better deployment of the resources allocated to the development and delivery of training content. To achieve this, we recommend:

   • Each division should have a national committee to provide direction for, and oversight of, both new member training and ongoing professional development for their members, with a mandate to:
     • Identify and prioritize members’ training needs.
     • Develop annual (or biennial) training and professional development plans, and accompanying budgets, for review and approval by the senior leadership of the IRB.
     • Work collaboratively with the other divisions’ training and professional development committees to identify common training needs among members that can be addressed using IRB-wide approaches and courses, as part of the planning process.
     • Monitor the development and delivery of member-led training and professional development programs, and modify planned approaches, as necessary.
     • Evaluate the effectiveness of the training and professional development provided in terms of its contribution to the achievement of the IRB’s mandate and strategic priorities.

     Each of these committees should consist of the DC and the regional ADCs, as well as senior Legal Services and adult learning advisors. The regional ADCs would each be responsible for leading the identification of members’ training needs in their regions, drawing on the results of region-level consultations, legal advisors would provide input regarding the substantive law knowledge required by members, and adult education would advise on the best use of experiential learning and evaluation practices. In the case of RPD, a senior advisor from the Research Directorate would also be beneficial, to provide input regarding training needs related to conditions in countries of origin of refugee claimants.

   • Divisional committees should be accountable to the senior leadership of the IRB. Each of the divisional committees should be responsible for the development and costing of annual (or biennial) plans for training and professional development, as noted above. These plans should provide an integrated approach to not just the design and delivery of new member training and ongoing professional development but also the systematic use of mentoring and observation or shadowing activities to support the ramping up of new members. The plans and proposed budgets should be presented to the Chair, Executive Director and DCs for review and approval. The committees should also be responsible for providing regular progress reports on the implementation of the approved plans and the results.
   • Design and implement a systematic process for identifying member training needs. The IRB should establish a consistent process, directed by the national training committees for each division, for identifying member training needs across all divisions and regions. This process should be based on consultations with members and the synthesis of needs identified through member monitoring and performance appraisals. The needs analysis process should be an integral element of the planning undertaken by each of the divisional training and professional development committees.
   • Implement a consistent approach to the assignment of mentors for new members and define the scope of the mentor role. Mentors are used to varying degree by all divisions; however the use of mentors is not institutionalized across the organization. The period in which mentors provide support to new members will need to be determined on a case-by-case basis that reflects the rate at which new members reach the point where there are considered to be able to make timely and defensible decisions on their own.

   Individual professional development plans for each member are also desirable, to guide the ongoing development of members. Accordingly, we recommend:

   • As part of the annual performance evaluation process, ADCs and coordinating members should work with their assigned members to develop personal professional development plans that address identified needs for further training and development. According to many of the interviewees the current evaluation process addresses individual training and development needs informally and should be strengthened. Consolidated information from these plans would also provide insights for the identification of training needs as part of the annual planning by the divisional training and ongoing professional development plans (as noted under the previous recommendation).


2. Improve the support for the planning, design and delivery of training programs
• With the exception of new member training for IAD, senior learning advisors from the LPDD have played minimal roles in the formulating the delivery of training courses and content to ensure that it is built on experiential learning principles. It is not clear from the interviews as to what role LPDD is currently expected to perform and whether it is sufficiently resourced to provide ongoing support for member training. Legal Services has recognized the need to apply experiential learning principles to a certain extent, but not consistently. Feedback collected via the member survey indicates that a substantial minority of members were dissatisfied with the provision of training on substantive law, and the reasons for this dissatisfaction mostly related to the selection of topics and the way in which material is presented.

We recommend that:

• The IRB should ensure that the recommended divisional training and professional development committees have access to senior adult learning advisors experienced in working with professional audiences. The role of the adult learning advisors in the divisional training and professional development committees defined in recommendation #1, above, will not to be to teach member training courses (unless they also have relevant subject matter expertise) but to contribute to the identification of training needs, provide guidance on the application of adult learning principles to the design of training programs and courses, and the evaluation of their effectiveness. These principles include such factors as maximizing opportunities to engage participants through the use of experiential learning methods (discussion, simulation, case studies, problem solving), ensuring content is meaningful to the participants, and directly applicable to their work. This expertise may already be resident in LPDD and, if so, suitably skilled advisors should be assigned to support the work of the divisional committees. If not, it will be necessary to secure the needed expertise by contracting services on an as-needed basis or by recruiting suitable personnel.

Additionally, our evaluation found evidence of a need for increased administrative/clerical support and translation resources to enable the timely production of training materials, especially, revised training content intended for national application. Nor were training and professional materials consistently retained and made available for further use across all regions and, as applicable, divisions. Consequently, we recommend that the IRB:

• Ensure that clerical/administrative support to enable the timely production of training materials is available to those developing and delivering training and that materials used in training are translated in a timely manner.
• Implement an information management process to retain training and professional development materials for follow-up access by members and/or repeat delivery.
3. Evaluate the effectiveness of current adjudicative training for members

   Evaluations of the effectiveness of training and professional development approaches and methods provide a critical feedback loop to enable the effectiveness of approaches to be assessed and evidence-based improvements to be made. We recommend the IRB:

   • Develop and implement a process for monitoring and evaluating member training to inform the planning work of national planning committees for each division. Key elements to consider include:
   • Development and implementation of standard feedback surveys to measure the reactions of participating members following each new member training course as well as structured professional development courses for other members.
   • Periodic online surveys of members regarding their satisfaction with the training received, the extent to which it enables them to perform their roles, and to identify possible gaps and weakness.
   • Reviews by DCs and ADCs of the extent to which members are meeting both quality and productivity expectations, and the extent to which training programs and courses are contributing to performance in these areas.

   ​There are also a number of areas where immediate evaluations would be beneficial: 

   • Undertake a formal review of the effectiveness of the re-designed RPD, IAD and RAD new member training to:
     • Assess the extent to which it is preparing the participating members to assume their roles.
     • Identify areas in need of improvement.
     • Assess the suitability of the length and ordering of the various components.
     • Assess the feasibility of streamlining the training provided to new members with relevant background experience (similar to what is currently applied in the RAD’s new member training).
4. Addressing unmet training needs

   Large proportions of the surveyed members indicated that they felt they receive too little training on developing their “adjudicative craft” skills: conducting hearings and making decisions, and managing caseloads and workflows. In recognition of this strongly expressed need we recommend that:

   • Divisional training and ongoing professional development committees incorporate plans for developing and delivering enhanced training approaches to better equip members to conduct hearings and make decisions, and manage their caseloads and workflows.

Appendix A – Research Matrix

Appendix B - Interview guide

The Immigration and Refugee Board of Canada (IRB) is conducting a study of its adjudicative training for new and existing members covering the period from March 31, 2013 to December 2017.

The purpose of the study is to review the design, development, and delivery of adjudicative training services provided to members in terms of their effectiveness, efficiency, best practices, and lessons learned. Adjudicative training consists of training and professional development activities that aim to improve the effectiveness and/or efficiency of members in fulfilling their role as decision makers. Non-member training and non-adjudicative training (e.g., language training, anti-harassment training) are outside the scope of this study.

The interviews are focussed on obtaining perspectives from senior management, members, and staff who are involved in the design and delivery of training at the IRB. With your permission, the interview will be digitally recorded to ensure that your responses are accurately captured. The information you provide will be summarized in aggregate form, meaning individuals will not be identified (or identifiable) in the reports that are generated from this research. If you are unable to answer a question, please let the interviewer know and they will skip to the next question.

Introduction

1. Please briefly describe your current roles and responsibilities as they relate to the planning, development and delivery of member training at the IRB.

Current state of LPD services

2. For any of the following activities that you may be involved with, please describe how they are currently carried out:
1. Identifying member learning needs;
2. Planning for training services;
3. Design and development of training services;
4. Implementation and delivery of training services; and
5. Monitoring and performance reporting on training services and their outcomes.
3. In your view, is there a sufficient level of interaction and collaboration between the various organizational units involved in the planning and delivery of member training activities (e.g., Legal Services; Learning and Professional Development Directorate (LPDD); IRB Divisional and Regional managers; professional development committees; and members)?
4. Are the roles and responsibilities among the various participants in the planning, development and delivery of member training clear and well understood?
5. Are the current processes and organizational structures for member training efficient (e.g., minimize duplication of effort, build on past experience, utilize best practices)?
6. Considering the current levels of resources (e.g., human and financial resources, learning tools and technology) for member training activities:
1. To what extent are they sufficient to meet current demand?
2. Do they provide the IRB with sufficient flexibility to address increases or changes in demand for member training services?
7. Based on your experience, to what extent do each of the following types of training provided to members equip them with the knowledge and skills required to make well-reasoned, timely and independent decisions?
1. Orientation and training for new members;
2. Ongoing training for current members relating to the substantive law and knowledge required for decision-making;
3. Ongoing training for current members relating to skills in conducting hearings and making decisions (e.g., reviewing evidence, writing decisions); and
4. Ongoing training for current members relating to managing caseloads and workflows (e.g., case management, time management)?
8. Do you believe members receive sufficient training to prepare and enable members to perform their roles? If not, in what areas do they need to increase their knowledge and/or skills?
9. A number of different formats are used to deliver member training, such as those listed below. To what extent do each of these delivery formats enable members to develop and sustain the knowledge and skills required for their roles? Which formats do members most prefer?
1. In-person group seminars and presentations (e.g., professional development days);
2. One-on-one (e.g., mentoring);
3. On-line webinars, video-conferencing; and
4. On-line self-study and research (e.g., accessing and reviewing information update bulletins and resource centre materials).
10. Who within the IRB is currently accountable for ensuring members receive the right training at the right time? If applicable, who should be accountable for ensuring member training is effective?
11. To what extent does member training align with the IRB’s:
1. Mandate, of making well-reasoned decisions on immigration and refugee matters efficiently, fairly, and in accordance with the law.
2. 2017-18 to 2020-21 strategic priorities:
1. Limit the growth of the refugee determination backlogs.
2. Eliminate the backlog of immigration appeals.

Best practices or alternatives

12. Are there alternative approaches to the planning, development and delivery of member training services at the IRB that could be more effective or efficient?
13. How might the IRB improve the processes, responsibilities, and lines of authority related to its training activities?
14. Are you aware of any best practices or lessons learned resulting from the IRB’s current approach to providing member training?
15. Do you have any other comments related to the IRB’s member training that have not been covered above?
    ​
    

Thank you for your participation.

Appendix C - Survey questionnaire

On behalf of the Immigration and Refugee Board (IRB), Kelly Sears Consulting Group and Corporate Research Associates are conducting an evaluation of the IRB’s adjudicative training for new and existing members. The purpose of the evaluation is to review the design, development, and delivery of adjudicative training services provided to members in terms of effectiveness, efficiency, best practices, and lessons learned.

Adjudicative training consists of training and professional development activities that aim to improve the effectiveness and/or efficiency of members in fulfilling their role as decision makers. Non-member training and non-adjudicative training (e.g., language training, anti-harassment training) are outside the scope of this study.

The objective of the survey is to obtain member perspectives on the delivery of adjudicative training at the IRB.

The survey should take approximately 10–15 minutes to complete. (Insert instructions for navigating through the survey questionnaire.)

The information you provide will be summarized in aggregate form, meaning individuals will not be identified (or identifiable) in the reports that are generated from this research. The survey will remain open until February 6, 2018.

If you have any questions or difficulties in terms of completing this online survey, please contact Peter MacIntosh at Corporate Research Associates Inc., either by telephone at 1-888-414-1336 or by email at pmacintosh@cra.ca

The survey is administered by Corporate Research Associates Inc., a survey research agency. For more information on Corporate Research Associates’ Privacy Policy, click here. This survey is registered with the national survey registration system. To verify the validity of this survey (number VWXYZ) with the Marketing Research and Intelligence Association, Canada’s national association of professional survey researchers, please click here.

Member profile

1. In which division do you currently serve as a member? SELECT ONE ONLY
• Refugee Protection Division (RPD)
• Refugee Appeal Division (RAD)
• Immigration Division (ID)
• Immigration Appeal Division (IAD)
2. In which region do you currently serve as a member? SELECT ONE ONLY
• Eastern region
• Western region
• Central region
3. In what year did you become a member of your current division?

2017	2012
2016	2011
2015	2010
2014	2009
2013	2008

Other: Please enter year _______ (enter year)

Throughout the survey, please focus on training you received in your current division during the past two calendar years (2016 and 2017).

Learning needs

4. To what extent are you satisfied that your division is effectively identifying the learning needs of its members?
• Completely satisfied
• Mostly satisfied
• Neither satisfied nor dissatisfied
• Mostly dissatisfied
• Completely dissatisfied
• Not aware that my division investigates the learning needs of its members
5. [DO NOT POST Q.5 IF 'NOT AWARE ...' IN Q.4] To what extent have you been consulted as part of the Division's process for identifying members' learning needs
• To a great extent
• To some extent
• To a limited extent
• Not at all
• Don’t know/Not sure
6. Do you have any suggestions for how your division can improve its processes for identifying members’ learning needs?
• No suggestions

Satisfaction with training content

7. Which of the following types of training provided by the IRB have you received in 2016 or 2017? Please select all that apply. DO NOT RANDOMIZE ORDER
• New member training
• Training for current members covering the substantive law and knowledge required for decision-making (received subsequent to new member training)
• Training for current members in conducting hearings and making decisions, e.g., reviewing evidence, writing decisions (received subsequent to new member training)
• Training for current members in managing caseloads and workflows, e.g., case management, time management (received subsequent to new member training)
• Other types of adjudicative training (please describe)
8. What is your level of satisfaction with the different types of training you’ve received in the last two years?

	Completely satisfied	Mostly satisfied	Neither satisfied nor dissatisfied	Mostly dissatisfied	Completely dissatisfied
a. New member training
b. Training covering the substantive law and knowledge required for decision-making
c. Training in conducting hearings and making decisions
d. Training in managing caseloads and workflows
Other training: [PIPE RESPONSE FROM Q7]

(Ask each of the following questions separately for each of the types of training received.) RANDOMIZE ORDER

9. To what extent do you agree or disagree that the new member training that you received …: RANDOMIZE ORDER OF STATEMENTS

	Completely agree	Mostly agree	Neither agree nor disagree	Mostly disagree	Completely disagree	Not applicable
a. Equipped you to make well-reasoned and timely decisions in your role as a member
b. Enabled you to manage your case load and workflow efficiently and effectively
c. Provided you with knowledge and guidance that could be immediately applied to your work as a member

10. (Ask only if there is at least one "Disagree" answer in Q.9a-c) What could have been done to improve or strengthen the new member training
11. To what extent do you agree or disagree that the training covering the substantive law and knowledge required for decision-making (received subsequent to new member training) …:

	Completely agree	Mostly agree	Neither agree nor disagree	Mostly disagree	Completely disagree	Not applicable
a. Equipped you to make well-reasoned and timely decisions in your role as a member
b. Enabled you to manage your case load and workflow efficiently and effectively
c. Provided you with knowledge and guidance that could be immediately applied to your work as a member

12. (Ask only if there is at least one “Disagree” answer in Q.11a-c) What could have been done to improve or strengthen the training covering the substantive law and knowledge required for decision-making?
13. To what extent do you agree or disagree that the training in conducting hearings and making decisions (received subsequent to new member training) …:

	Completely agree	Mostly agree	Neither agree or disagree	Mostly disagree	Completely disagree	Not applicable
a. Equipped you to make well-reasoned and timely decisions in your role as a member
b. Enabled you to manage your case load and workflow efficiently and effectively
c. Provided you with knowledge and guidance that could be immediately applied to your work as a member

14. (Ask only if there is at least one “Disagree” answer in Q.13a-c) What could have been done to improve or strengthen the training in conducting hearings and making decisions?
15. To what extent do you agree or disagree that the training in managing caseloads and workflows (received subsequent to new member training) …:

	Completely agree	Mostly agree	Neither agree or disagree	Mostly disagree	Completely disagree	Not applicable
a. Equipped you to make well-reasoned and timely decisions in your role as a member
b. Enabled you to manage your case load and workflow efficiently and effectively
c. Provided you with knowledge and guidance that could be immediately applied to your work as a member

16. (Ask only if there is at least one “Disagree” answer in Q.15a-c) What could have been done to improve or strengthen the training in managing caseloads and workflows?
17. To what extent do you agree or disagree that the other adjudicative training provided by the IRB (received subsequent to new member training) …:

	Completely agree	Mostly agree	Neither agree or disagree	Mostly disagree	Completely disagree	Not applicable
a. Equipped you to make well-reasoned and timely decisions in your role as a member
b. Enabled you to manage your case load and workflow efficiently and effectively
c. Provided you with knowledge and guidance that could be immediately applied to your work as a member

18. (Ask only if there is at least one “Disagree” answer in Q.17a-c) What could have been done to improve or strengthen the other adjudicative training provided by the IRB?
19. How would you assess the amount of training provided in each of the following areas?

	Completely agree	Mostly agree	Neither agree or disagree	Mostly disagree	Completely disagree	Not applicable
a. New member training
b. Training covering the substantive law and knowledge required for decision-making
c. Training in conducting hearings and making decisions
d. Training in managing caseloads and workflows

20. (Ask if “Too little” selected for any content types in Q.18.) What specific subject areas would you like to receive more training on?
21. (Ask if “Too much” selected for any content types in Q.18.) What specific subject areas would you like to receive less training on?

Satisfaction with delivery formats

22. Please indicate how satisfied you are with the various delivery formats used by the IRB to provide adjudicative training, in terms of their suitability for communicating the content and guidance required to do your job: ROTATE STATEMENTS, EXCEPT ALWAYS KEEP STATEMENT ‘e’ LAST

	Completely agree	Mostly agree	Neither agree or disagree	Mostly disagree	Completely disagree	Not applicable
a. In-person group seminars and presentations
b. One-on-one (e.g., mentoring)
c. Training in conducting hearings and making decisions
d. Training in managing caseloads and workflows

23. Thinking of all the adjudicative training you have received over the past two years, please rate your overall satisfaction with each of the following aspects (if an item does not apply, please check “not applicable”): (Rotate order of presentation)

	Completely satisfied	Mostly satisfied	Neither satisfied nor dissatisfied	Mostly dissatisfied	Completely dissatisfied	Not applicable
a. Opportunities to ask questions and discuss content with presenters
b. Opportunities to discuss content with other members in attendance
c. Opportunities to practice the application of content, e.g., through group exercises and case study simulations
d. Quality of supporting handouts and reference materials
e. Quality of instructors and their command of the subject matter
f. Provision of training in the official language of choice
g. Opportunities to provide feedback on the appropriateness and quality of the training I received
h. Suitability and reliability of technologies used for content delivery, e.g., online video conferencing, in-person audio-visual presentation equipment

24. Which aspects of the different formats used to provide adjudicative training (in-person group sessions, one-on-one mentoring, online webinars and videoconferences, online self-study) do you find to be:
1. Most effective?
2. Least effective and need to be changed?
25. Finally, do you have any other suggestions or comments regarding the planning, development and delivery of training to members?
• Yes (please describe)
• No

Thank you for participating in the survey.

Appendix D - Comparative analysis interview guide

Interview guide: comparative analysis of the [Social Security Tribunal]
[National Judicial Institute]

Background

The Immigration and Refugee Board of Canada (IRB) is conducting a study of its adjudicative training for members/judges covering the period from March 31, 2013 to December 2017. Kelly Sears Consulting Group is conducting this evaluation on behalf of the IRB. For further information, please contact: Glenn Ng, Performance, Risk and Evaluation Coordinator: Glenn.Ng@irb-cisr.gc.ca.

The purpose of the study is to review the design, development, and delivery of adjudicative training services provided to members/judges in terms of their effectiveness, efficiency, best practices, and lessons learned. Adjudicative training consists of training and professional development activities that aim to improve the effectiveness and/or efficiency of members/judges in fulfilling their role as decision makers. Non-member training and non-adjudicative training (e.g., language training, anti-harassment training) are outside the scope of this study.

The evaluation involves several data collection activities, including interviews throughout the IRB, an online survey of IRB members/judges and a comparative analysis of the approaches to providing training and continuing education at a number of similar organizations, including the … .

The information you provide will provide input to the assessment of certain evaluation issues, and will exclude any confidential or sensitive information. A copy of our draft comparative analysis of the …’s approach to member training will be provided to you for review.

Guiding questions

1. Please provide a brief overview of the ...'s:'
1. Main activities
2. Organizational structure
3. Number of employees
4. Number of members/judges supported
5. Annual expenditures
2. Please describe the key features of the …’s approach to the provision of:
1. Initial orientation and training for new tribunal members/judges.
2. Ongoing training and continuing education for current tribunal members/judges regarding:
1. Substantive law and knowledge required for decision-making
2. Skills in conducting hearings and making decisions (e.g., reviewing evidence, managing hearings, writing decisions)
3. Skills in managing caseloads and workflows (e.g., case management, time management)
3. Other support for the training and development of tribunal members/judges, including (but not limited to) observation/shadowing of experienced members/ judges by new members/judges, and mentoring of new members/judges by ex-members/judges.
3. How does the … undertake each of the following elements of its training and continuing education services? What roles do experienced tribunal members/judges, legal advisors, adult education specialists and other advisors play in these processes?
1. Identifying members’/judges’ learning needs
2. Program and course planning
3. Design and development of content and delivery methods
4. Delivery
5. Monitoring, evaluating and reporting on activities and outcomes achieved.
4. Does the … prepare annual or longer-term strategies and plans to guide its provision of training and continuing education for new and existing tribunal members/judges? If so, what is the process for preparing these strategies and plans, and informing members/judges of the planned training and education activities?
5. Who within the … is accountable for ensuring members/judges receive the right training and continuing education at the right time?
6. What is the organization structure for the planning, development and delivery of training and continuing education services at the …?
7. Approximately how many people (FTEs) are employed in these activities? How many different training sessions are delivered per year?
8. To what extent does the … utilize online methods – such as group webinars and videoconferences or online self-study – as an alternative, or extension, to face-to-face methods? What factors are considered when choosing between face-to-face and online delivery methods?
9. Do the training and continuing education needs and priorities of members/judges vary between provinces and territories (or regions)? If so, how does the … ensure these local needs are identified and addressed, in addition to addressing the organization’s national priorities for training and education?
10. What would you consider to be the best practices and lessons learned with the …’s provision of training and continuing education to tribunal members/judges?
11. Have there been any recent evaluations or reviews of the effectiveness and/or efficiency of the …’s training and continuing education activities? If so, could you provide us with a copy?
12. Finally, do you have any other comments related to the …’s training and continuing education that have not been covered above?

Thank you for your participation.