 |
 |
 |
 |
What's New | ||
|
Clinical Services | ||
|
Training and Consultation Services | ||
|
Fa.S.T. | ||
|
Get Together FaST | ||
|
Publications | ||
|
Products | ||
|
The Notice Board | ||
|
Links | ||
|
|||
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
The project commenced in mid 1997, with planning, recruitment, establishment of advisory
groups and commencement of consultation with the field. Training was conducted between
February and November 1998. The project was implemented in three 'streams': Adult, Aged
and Child and Adolescent Mental Health (CAMHS) Services, each having designated training
consultants and developing related, but distinct, materials and training experiences.
A full description of the aims, strategies, consultation processes and training programs
is available in the main Project report available from the Bouverie Centre and the Mental
Health Branch. Please contact Brendan O'Hanlon. Email:
b.o'hanlon@latrobe.edu.au.
We know of no validated measure of 'family sensitive practice', yet a key evaluation
question is the extent to which participants may view families and family work more
sensitively or more favourably as a result of the training. This evaluation therefore
built upon the work done previously in the Northern Area Pilot Study (Farhall, 1996).
In that study, a questionnaire was devised for workers in adult mental health services
(the Opinions About Family Work Scale) drawing upon the published literature on
professionals' attitudes towards families and also utilising themes raised in a
focus group of family carers and advocacy workers. A key difference in this study
was the inclusion of staff from Aged Mental Health Services and Child and Adolescent
Mental Health Services. After attendance at the Aged Advisory group and discussion of
the adult version with the trainers, an expansion of the adult version was considered
to be equally relevant of carers of aged consumers and adult consumers, thus the same
expanded version was used in both streams (See Appendix). This 'Opinions About Family
Work Scale - Adult/Aged Services Version' included 26 of the original 30 items,
supplemented by 14 items suggested by a further review of the literature, discussions
within Reference Groups and feedback from the trainers. These new items included
statements addressing difficulty or complexity of work with families, and included
acknowledgement of issues facing carers of aged consumers. The CAMHS Version had 22
items in common with the adult version and an additional 16 items which introduced
themes of treatment needs of the parents, emotional disturbance in the family and the
family's need to be informed about services for their child or adolescent. These items
were derived from a review of the literature, and from discussion with the CAMHS stream
trainers.
For this study, measures of attitude were derived in two ways for each version of the
Opinions About Family Work Scale. A simple 'Pro-family' score and 'Difficulty of
family work' score were computed by totaling relevant item ratings. Taken at face
value, most of the individual questionnaire items were either positive or negative
statements about families and family work. Ratings on these were combined for each
individual to give a 'Pro-family' score. The 'Difficulty score' was also computed by
totaling scores on items describing difficulty and complexity in working with families
(See Section 2.5 for list of items allocated to Pro-family and Difficulty measures).
The evaluators followed up both the topics and scope of these project proposals and
the proportion of proposed projects actually implemented. Information about the
projects was gathered from the 'Managers/Team Leaders/Coordinators Questionnaire'
and the Project Brief Profile sheet which facilitators completed at the post
training consultation with each area mental health service.
Although participant feedback questionnaires requested information from staff about
numbers of family members seen over the past week, these data were known at the
outset to be a very limited opportunity to pick up any change in service patterns,
for the following reasons. These questionnaires
Information about staff contacts with family members was made available by the
Department of Human Services from the PRISM database. To preserve confidentiality,
identifying information was anonymised prior to the data file being made available.
The data file included information about the number and nature of contacts recorded
by adult and aged services staff with their clients and family members, both together
and separately. The data comprised a one-month period prior to implementation of the
FaST training and a one-month period following the training. PRISM data for CAMHS was
incomplete and therefore not able to be used. Comparison data for the same months of
the previous year were also examined to identify any pre-existing trends in family
contacts.
Participating carers and consumers were contacted by the research assistant for a
structured feedback interview regarding experiences of participation, satisfaction
with the project and recommendations for future service development (See Appendix
for Questionnaires).
The full report is available from the Bouverie Centre. Please contact Brendan O'Hanlon. Email:
b.o'hanlon@latrobe.edu.au.
An overview of the "Get Together FaST" project and its evaluation
John Farhall
La Trobe University
Psychology Department
The "Get Together FaST" (GTF) project
The 'Get Together FaST' (GTF) Project was a large scale training and service development
initiative. Its general aim was to further the implementation of Victorian Mental Health
Branch policy as outlined in the 'In Partnership' document (Human Services Victoria,
1996). More specifically, its purpose was to assist mental health services staff to
work more collaboratively with families and other carers of those with mental illness
and disability.
Evaluation strategy and methods
Introduction
The evaluation strategy for FaST aimed to assess any impact of the project at multiple
levels ranging from the intra-personal (did the project change staff beliefs about
working with families?) to the statewide (did the project lead to changes in patterns
of contact with families?). This section outlines the rationale and methods for the
main evaluation components. Most measures were collected via Pre-training and Post-
training questionnaires completed by participants in the 44 training programs. In
addition, the project devised questionnaires (or structured interviews) for other
parties such as managers, carers, consumers and the trainers. All questionnaires
are reproduced in the Appendix.
Participation
Participation in any training and service development project is a prerequisite for
success. Without appropriate levels and representativeness of participants, the goals
of the Get Together FaST project would be unlikely to be achieved, even if the plans
and training content were excellent. Thus, participation was evaluated by:
Feedback about the training from participants
A simple measure of success of a training project is its acceptance by, and impact
upon, the target participants. Clearly, if the training process does not engage
participants, or if the content is perceived to be unsatisfactory, benefits of
training are less likely to emerge. The Post-training questionnaires (See Appendix)
included questions seeking overall satisfaction ratings, opportunities to comment on
best and worst aspects of the program and feedback ratings on key elements of the
program.
Changes reported by participants
A simple way to measure of the impact of an intervention is to ask the participants to
rate themselves on relevant outcome variables before the intervention and then to rate
themselves again after the intervention. It is then possible to determine whether any
change in rating is likely to be greater than that expected by chance. This method was
used to measure self-rated changes in family sensitive practice and self-rated
experience, skill and knowledge. For each stream, the 'Self-rated family sensitive
practice' score was derived by summing ratings on items which asked the extent to
which participants currently put into practice the objectives of the training.
These objectives were clarified with the trainers and formulated into statements
for the questionnaires. The skill, knowledge and experience ratings enabled us to
study the number of people who changed their ratings following training.
Attitudes towards working with family carers
The Get Together FaST training aimed to influence opinions and attitudes about families
and family work. Within the broad context of government policy promoting "partnerships"
with family carers ((Human Services Victoria, 1996)), the training encouraged sensitive
understanding of the point of view of family carers. This was addressed at many levels;
the inclusion of family carers in the training program, the language used by trainers,
the concepts and models utilised (particularly the 'trauma model'), the emotive
messages of the FaST videos and so on. The intention was to encourage a higher
level of 'family sensitive' practice in attendees, at least in part by promotion
of more family-sensitive attitudes. It was therefore a central aim of the evaluation
to determine whether any intended attitude change occurred.
The second method of attitude measurement was based on Factor Analysis of the
underlying themes in each questionnaire. Data sets from each version of the
questionnaire were subjected to a principal components analysis, with four
underlying attitude factors identified in each case. Each person was then
given a score reflecting their level of attitude on each of the four identified
factors.
'Objective' indicators of change in participants and services
Self-report measures are a useful source of information about acceptability of an
intervention and about the participants' perceptions of change. Attitude measures
are a more powerful way of determining whether change may have occurred. However,
supplementation of these by use of sources other than the participating staff is
desirable, to bring in 'objective' sources of evidence for change.
Managers/team leaders/co-ordinators
One source of more objective observations of change is the superiors of the staff
attending the training. To tap this source of observation, a 'Managers/Team
leaders/Co-ordinators' Questionnaire was devised. Consultation with the local
convenors for each area training program provided a mailing list of managers,
team leaders and coordinators of participants who had attended GTF. These managers,
team leaders and coordinators were then sent a questionnaire and invitation to
participate in the GTF evaluation. Some had attended the training and some had not.
The managers questionnaire (See Appendix) primarily sought information from managers
about what feedback had been given by staff about the training, as well as
observations by the manager about how GTF may have affected their organisation.
Project Briefs/Proposals
The GTF project was conceptualised as a service development initiative as well as a
training program. Syndicate activities were the main element of the service
development aspect of GTF, although the pre and post-training consultations also
included discussions beyond the domain of training. Training participants formed
syndicate groups that developed a project brief or proposal as part of the training
program. Projects were to be aimed at furthering Family Sensitive Practice in the
workplaces represented in the syndicate.
Statewide Contact Data
Increases in the numbers or duration of contacts with family members was not a primary
outcome intended by the project, but was considered to be a likely outcome on the basis
of the Northern area pilot program.
For the above reasons, these self report data were supplemented by information from the
statewide database of mental health service activity (PRISM).
Involvement of carers and consumers
The government policy that led to funding of this initiative promotes partnership with
families in both planning and delivery of services. Although the content of the
training was intended to reflect this aim, the processes of the project were also
expected to be inclusive of families. Part of the evaluation therefore focussed
on the degree of involvement of family carers in planning and training delivery,
and their experiences and recommendations for the future. Participation of
consumers is also an established element of government policy for mental health
services. Their participation in this project was also covered by the evaluation.
Feedback from trainers
The Bouverie staff and external trainers who implemented the program are in a unique
position to comment on the strategies and effectiveness of various elements of the
project. The research assistant conducted structured interviews with each trainer
at the conclusion of the project.
Data collection approvals
As an externally funded evaluation project conducted by La Trobe University staff,
the evaluation required, and obtained, approval from the University's Human Ethics
Committee. In addition, all participating mental health service organisations were
contacted to seek appropriate approvals for the evaluators to invite participation
of training participants and managers in the evaluation. This led to the project
obtaining approval via Research Ethics Committees in two organisations and approvals
via Quality Assurance Procedures in 25 organisations. Approval for the voluntary
participation of training participants from non-government organisations (NGOs)
was covered by approval of the area mental health services as auspices for each
training round. Organisational approval for the participation of NGO managers
was usually taken by the senior manager of the service on receipt of the invitation
to participate.
Summary of measures and evaluation procedures
Pre-training Questionnaires
Three separate but overlapping questionnaires were developed for each stream and
completed by participants on arrival at the first training session. The
questionnaires covered background information (profession, service type, etc.),
self-ratings of skill, knowledge and experience, self-ratings relating to family
sensitive practices (objectives of training), self-reported contacts with families,
and the appropriate version of the 'Opinions About Family Work Scale'.
Post-training Questionnaires
The 'Post-training Questionnaire' sought feedback about the training program
(overall rating, ratings of key elements) as well as repeat ratings of skill, knowledge
and experience, self-ratings relating to family sensitive practices (objectives of
training), self-reported contacts with families, and the appropriate version of the
'Opinions About Family Work Scale'. This questionnaire was given to all participants
who attended the last module of training. In addition, people who were not present for
the last session were also sent a questionnaire, although the return rate for this
group was lower.
Managers/Team leaders/Co-ordinators Questionnaires
These were sent to the managers of all training course participants within a two month
period after the training had been completed in each respective area. The
questionnaire sought information from managers about any feedback staff had
given about the training, as well as observations by the manager about how the
FaST project may have affected their organisation.
Contact data from the PRISM database
Anonymous client and family contact data was made available by Mental Health Branch.
Analyses sought to determine whether, at a statewide level, the number or proportion
of contacts by staff with family members rose between February and September 1998,
the period during which most of the training and follow-up interventions occurred.
Follow-up of Project Briefs
Six to eight weeks following completion of training in an area mental health service,
the training consultants met with the area co-ordinator (and sometimes other
senior staff) to review the FaST project and to discuss any further needs. At
each of these meetings, the training consultants brought along a list of the
projects initiated by participants from that area mental health service. The
current status or outcome of these projects was recorded and compiled for
evaluation.
Interviews with carers and consumers who participated in GTF
All carers and consumers involved with the FaST project were invited to participate
in a structured feedback interview covering experiences of participation, satisfaction
with the project and recommendations for future service development.
Interviews with GTF training consultants
Informal interviews were conducted individually with the trainers who implemented
the Get Together FaST project.
Statistical data and conventions
Unless otherwise stated, all relevant tests are two-tailed and a p value of less than
0.05 is accepted as significant. Where multiple independent statistical tests occur,
there is a danger that some will achieve statistical significance by chance. To guard
against making Type I errors in these situations, a Bonferroni adjustment was made to
the acceptable p value by dividing the p value by the number of independent
significance tests. This adjusted threshold level of statistical significance is
stated when required.
When conducting ANOVAs, subgroups of staff were not included in analyses if the number
of participants fell below eight.
 top |
Community Services |
Clinical Services |
Academic Services The Bouverie Centre: Victoria's Family Institute 8 Gardiner Street, Brunswick 3056 Phone:9385 5100 | Fax:9381 0336 | email:bouverie.centre@latrobe.edu.au |
top |