An International Database and eJournal for Outcome-Evaluation and Research


Moving from a general to a personalised model for the evaluation of outcomes: Perspectives from international research


Meaning of the Conference. The international conference on outcome based evaluation on child and family services could be seen as a successful experiment. When we were planning it, we wondered, as usually happens, which issue would be so relevant to attract so many people from a number of countries.

Often, it is easier, and somehow more natural, to consider the needs of children and their families, their problems, the reasons behind their sufferance, the many obstacles interfering with the opportunity for a normal growth, for developing peculiar skills and for healthy living in their life space.

But when doing this, challenges and difficulties are overwhelming. For instance, problems such as child maltreatment, abuse, delinquent behaviour, social exclusion, problematic behaviours, deprivation and many others should be addressed. In the real world, on these topics, there are more problems than solutions, even if many people and different organizations have been dealing with researches on these themes for many years.

For the conference, we chose to focus on the point of view of the children and parents in difficulty, living everyday in the conditions listed above. How could we do this? We considered the articulation and complexity of children needs, without putting them in an order of importance. One thing is clear: those taking care of children needs, because of their profession or their voluntary service, are interested in overcoming their obstacles and difficulties, and in opening new opportunities which before were unseen. Those questions are the same addressed by outcome evaluation. In fact, as compared to process evaluation, and evaluation of output, outcome evaluation should reach the heart of the issue, investigating whether the fundamental needs and rights are met, and, if so, with which results.

In case of children and adolescents problems, most of them come from the negation of fundamental rights, which are necessary to their growth, personal and social develop­ment, and their successful development towards adulthood.

Outcome evaluation, trying to reach the heart of the issues, forced us to ask ourselves if what we have been doing has been useful, effective and has given answers to the problems.

We were not only asked whether those helping have worked correctly, has followed the procedures, and has used the available resources.

Effective outcomes cannot be reached only through the active involvement of professionals, as well as the availability of resources does not imply successfull outcomes. Many examples are available on the contrary. Being this the case, many problems would be solved or would have the answers we have long looked for. In this sense, outcome evaluation is not only a technical and methodological issue, but also strategic and ethic one. It sets the reasons for effectiveness above the reasons, also important, for quality, for technical and professional pertinence, for an appropriate use of resources.

The effectiveness of helping process needs all those elements but it does not depend on those elements only. In this sense, we need to understand what makes the difference, and we do not know yet, that is what makes it possible to go from the necessary but not sufficient conditions, to those conditions able to reduce and overcome the problems.

These questions are the premise of the experiment we are living at the moment. When putting these questions at the centre of a conference, promoting the meeting between many different countries, different cultures, different welfare systems, we did not know whether those questions would challenge others as well. The answer is available in this book and, mostly, in the amount and importance of contributions, which has given and will give a lot to the researches on the improvement of outcome evaluation, and more than this, to make more effective the work of those (professionals and services) help children, adolescents and families in difficulty everyday.

I will go now to the specific problem at the centre of "the conference", that is how outcome evaluation could be a difference, or a paradigm shift for looking for new solutions, more able to provide an answer to the many problems on the field.

Conditions for shifting the paradigm. Outcome evaluation aims at measuring and evaluating the quantity of benefits obtained by individuals (children, young people, families) after the provision of social, health educational services. There are a lot of reasons for demanding for services and conseguently for providing services: how can we provide a stable and effective foster care, given that sometimes this is not the case? How to support foster families in their task? How can we help multiproblematic young people with severe problems that require an intensive us of resource? How can we manage at best and support young people leaving care? How can professionals' and children's views be matched, focusing on difficulties as expressed by children and parents? How to develop effective strategies for adolescents? How can we organise residential care in a way that it becomes an actual solution, not only a setting for containing disturbed behaviours or a setting in which more problems can arise? Are there any alternatives to the intensive treatment? Why children with mental health difficulties are not considered for their health needs? How can we better consider the at-risk behaviours in children and adolescents and activate early interventions? How to develop the necessary competences for providing effective services taking into account different cultures? How to improve parenting competence, given that many problems derive from this lack of competence in young parents? How can we help children and families with disabilities without stigma ?

All these questions (and other ones) require that, on one hand, services have the professional capacity to meet the needs and, on the other hand, users have the capacity to express their potentialities. Matching these services' and users' capacities allows us to define shared objectives and results to reach.

Outcome evaluation is rooted on this combination between services and users. In particular, it focuses on the time and the way needs are met and problems find solutions able to reduce their suffering, empowering users' capacity, their responsible choices, inside their life and relational environment.

In the evaluation of outcome, there main interest is for the gain of well-being, healthy living, social integration, development of potentialities, reduction of obstacles that impede to express fully the personality of individuals.

Partly, these results depend on the capacity and quality of the professional actions, even if this is not enough for explaining them. In other words, the quality of process (professional, organisational, etc.) is a necessary condition but it is not sufficient. In techni­cal terms, we can say that good care provisions as well as the quality of interventions (output) are not enough for guaranteeing good effects (outcome) for the person and/or the family in tune with their expectations. This happens every time that the situation doesn't show significant changes in the need condition.

Therefore it is very important to highlight sistematically the difference between process evaluation and outcome evaluation, keeping in mind that these two evaluation and not consequent and isomorphic. This is well known by those practitioners working in the direct practice; they daily face the difficulty in providing good interventions, without of obtaining good outcomes.

Research areas. Papers in this book give a map of problems and challenges that today many of us are facing, thanks to the committment and efforts of professionals, services and countries. We could say (taking into account the outcome logic) that this map describes the main conditions of suffering and exclusion that we will face in a multi-cultural and transnational perspective.

A difficulty often encountered by research and practice is the cronic distance and in many cases the separation between research and practice, theory and experimentation, laboratory and every day contexts, research and dissemination of professional outcome-based choices.

The separation between traditional research and experimental research in real life nurtures a schizophrenia between thinking and working, given that most research devote itself to the analysis and description of problems and not enough to the experimentation and validation of new solutions.

In order to fill this gap, it is necessary not only to observe, describe, and explain. It is necessary to make more efforts and investments towards research and validation of effective solutions, able to facilitate professionals' choices in the best interest of the children and/or families. In order to reach this aim, it is necessary to build experimental settings and designs finalised to assess the impact of the solutions that we expect effective. In this view, experimental designs can be obtained for example thinking and developing further the meaning of experimental and control groups, given that in a transnational perspective this difference can be enlarged and better managed.

It is necessary to devote our attention to the meaning of experimental research on outcomes, considering not only the "functioning" of this process but mainly the indicators of need reduction and the indicators that suggest that the need has been overcome. This means that we need to work on the direct effectiveness, not as a proxy of something else, shifting from the good practice to the logic of best practice and for this reason outcome based.

We need bridges as collaborative infrastructures between researchers and professionals, theory and practice, methodological reflexion and operational paths, considering from a different point of view the challenges that we all know.

These are not easy challenges, given that it is easier and seductive to observe and describe than experiment, reducing the complexity and accepting to compromise the conditions of knowledge and therefore the possibilities of helping. For example, it is not enough to gather causal associations and explicative inference on what works or not. It is necessary to use them for going further and develop experimentally solutions for helping more and better.

The research on outcome evaluation must go beyong the Hercules Columns, facing the open sea and using more determination for developing an effective support. Only in this dimension (not only ethical hope but professional possibility) theoretical reasons and values can meet, work on the same field and engage joint challenges.

We will mention some challenges based on the shift from "evidence based" research to "outcome based" research. Both share many positive reasons for existing and collabora­te, but the first one focuses on our scientific knowkedge and the way to use it; the second one focuses on what we don't kow more in order to better orientate the choices, not only based on the reasons of appropriateness but on the potentials of new knowledge for more effective help.

Methodological and strategical questions. The use of quantitative and qualitative eva­lua­tion strategies increase the possibility to see if we met the needs, observing and gathering information regarding changes so as they can be recognised by different subjects. From here, the problem is how to observe changes that - for their nature - are rooted in different dimensions and domains: physical, functional, cognitive and behavioural, socio-environmental and relational, values and believes domains.

Technically speaking, the solution called S-P, "polar scheme", is providing us with promising results especially on multiproblematic situations. Also it facilitates the comprehensive vision on the needs represented by the distance from the center (that represents the problem) (Vecchiato et alii, 2005). More the polar scheme is concentrated in the center (problem) and more the difficulties are high. When the map enlarges towards the external circle and more the situation improves and allows the measurement of improvements, the effectiveness  and the possibility to leave the care. Furthermore, it gives an immediate vision on many different dimensions and the way they can be represented "together" and "at the same time", even if they come from different profes­sional views and responsibilities. It is a tool for partecipatory evaluation, that is an evaluation in which different views (including professional, user and his/her family) meet together.

Research data suggest that the polar scheme can facilitate need assessment and it can allow to share responsibilities in order to develop the personalised care plan.

With the polar scheme it is possible to shift from a vision based on x-y axis, such as the baseline (Blyte e Tripodi, 1993) to a representation that takes into consideration dimensions composed of multiple variable and multiple factors and this will affect the decisions. We are deeply involved in this idea, in order to bring results and experimental solutions that are focused on a theoretical and operational model called "PERSON" that means "Personalised Environment for Research on Services, Outcomes and Needs". This model reminds us of the daily challenge of working for helping people and with people. For this reason it "contains" different dimensions:  the person, his/her context, resources (services, people, local community), needs to face, expected results, the commitment and efforts for developing new solutions thank to outcome based research.

For this reasons, this model tries to integrate multiple factors with the complexity of problems we have to face. At the same time, it reminds us that the centre of our work - the person - force us to do our best for personalising care. Advantages are relevant for service users (in our case, children and their families) but also for professionals. In this respect, outcome evaluation is a humanisation factor because it sistematically reminds us the final goal: to know the health and well being benefits that we reached in terms of how many and which kind of benefits for children and families. We can say that an unifying element is represented by the evaluation person-based and we can use this element as a general criterion for further research on evaluation strategies, better distinguishing different characteristics (process, output, outcome).

Another example of possible research development refers to a more systematic use of methodologies and transforming technologies, as in the S-P model. Transforming methodologies allow us to represent problems in a non reductive way, to describe expected results, to define possible choices and decisions that are necessary to reach the expected results. On this basis, we could obtain a high benefit from the technical possibility to compare the differences between expected outcomes and obtained outcomes (from professionals and services in our country and/or in different countries), to understand the reasons, to improve them and to enlarge the outcome based knowledge.

For example, the systematic analysis on differences between expected and obtained results (as we are doing) represents a basis for researching outcomes more valid and consistent, that means with a higher helping capacity, and reducing the variance between expected and obtained outcomes, as we can represent in the following figure.

A third example comes from the collaborative research inside the International associa­tion on outcome-base evaluation and research (iaOBERfcs) that is trying to develop a common platform for comparing services, costs and outcome indicators. It looks like an impossible task, given the huge differences in each country and among countries.

Also in this case, the transforming method can be a solution, that is the use of a classification code that is obtaining good results in terms of comparison between costs and services. For this reasons we are developing research conditions for including in this work also outcome indicators. The common platform for the classification (Canali, Vecchiato, 2007) is based on three macro variables as well as the needs. The macro variables are then more specified taking into account their similarities and species: this allows comparisons and sharing outcome based knowledge among different countries and/or  regions (as we are doing comparing different Italian regions).

Conclusions. The evaluation of outcome as described in this paper represents a priority for those involved in the humanisation of services. Such a strategic question for helping children and families represents a priority and for this reason it cannot be left only to professional choice but also political and strategical choice. Today this question doesn't have adequate answers. The following question considers the issue: should this option be guaranteed by professionals and agencies as a technical and ethical priority? This idea of outcome based evaluation increases the possibility to meet effectively the needs, to overcome them, like a gold standard and not only a recommend­ed practice.

If we agree on this idea, we should include it in the basic levels of care to be provided as a right for the person. If so, it should become a political, professional and ethical priority, to be implemented by those involved in children and family services in different welfare system.

Key references

Blythe, B.J., Tripodi, T., Fasolo, E. & Ongaro, F. (1993). Metodi di misurazione nelle attività di servizio sociale a contatto diretto con l'utenza. Padova: Fondazione Zancan.

Canali, C. & Vecchiato, T. (2007). "Multiple problem children and families: findings from Italy". in Grietens, H., Knorth, E., Durning, P., Dumas, J., eds.. Promoting competence in children and families. Scientific perspectives on resilience and vulnerability. Leuven: Eusarf.

Vecchiato T., Canali C., Pompei A., Bezze M. (2005), The polar scheme: a strategy for outcome-evaluation. A paper presented at the Stockholm Workshop on Assessment Instruments in Social Work Practice and Research, Ims, National Board of Health and Welfare, May 23-24, 2005.

Zeira A., Canali C., Vecchiato T., Jergeby U., Thoburn J., Neve E., Il lavoro sociale basato su prove di efficacia con minori e famiglie: prospettive internazionali, Studi Zancan 1/2007.

Contacts: Tiziano Vecchiato, Director - Fondazione Emanuela Zancan onlus, Via Vescovado, 66 - 35141 Padova Italia, E-mail: tizianovecchiato@fondazionezancan.it, Tel. 0039 049 663800.



© copyright 2023 Outcome-Network.org all rights reserved, in partnership with FondazioneZancan | iaOBERfcs | read the legal notice.