Notes on how to evaluate Educ RefNOTES on HOW TO EVALUATE EDUC REF? Hugo Zegarra This article contains my notes on Wiley’s article. SEE: Systems Research and Behavioral Science, May-June 1998 v15 n3 p209(7) Before making a summary of John Wiley article “Evaluating Educational System Design” (1998), I want to call the readers attention to the following arguments: NOTE 1 1. “Educational systems does not repond to the need of the learners”. This is one of the official stories about educational crisis. If you can change the word “learners” for society, parents or communities and you have the rest of similar hypocrit and coward arguments for reforming the school system. It is easy to talk in the name of “learners” because it refer to children and do not have voice. If they say that education do not repond to the “needs” of the society, parents or communities, there will be many voices that will jump to responde or ask questions such as Education for what development?. To export raw materials without adding value to them (mineral and agricultural products), for neoliberal economy we don’t need to make any school reform. All that we need is primary school, that is basic math, reading and communication skills to operate machines under the supervision and control of bilingual foreign managers. To pack saparragus or load trucks with gold, silver and other minerals we don’t need secondary education, this should be transfered to private institucions, they argue. If the State cannot afford to finance primary school, if it is in real bankcrupcy, then primary education should be transferred to local municipalities and since most of them do not have resources either, privatization is the solution. That is the way in which most central States are addressing the issue crisis of the education system. An honest argument about this crisis is to say that States have other obligations or prioritees to attend than providing enough resources to education, but the servants of the Central State are not pay to be honest. A suttle though direct way of addressing the issue would be not only talking about insufficient resources but the miss-management of the few resources given to education. That argument set the basis for the next one, the transfer of financial responsabilities to local communities or private institutions. We have to decentralize the system, that is the common feature and official way of thinking about education reforms nowadays. Does it mean that decentralization will solve the problem of insuficient resources in countries and local communities where more than 50% of their inhabitant suffer accute and chronic poverty?. If privatization of basic education is the hidden agenda for decentralization, how this policy is going to prevent a widen gap of intra-regional inequalities between poor and rich communities?, and what about urban-rual national gaps or differences?. How to explain the fact that a central State that tacitly admit having not power to set up national an regional “equalizing funds” to cop with current growing inequalities, will lather on have such a power?. The reader should keep in mind that -if decentralizacion implies a real transfer of economic and political power to region and municipales- the power gained by decentralized units of the State that will be at expenses of centralism. This means that later on the Central State will have less power to set up such “equalizing funds” to stop the gaps of intra-regional and national inequalities. These are some of the incositences of central state bureucrats regarding education reforms to cop with the current crisis of the educational system. It is paradogical that is the central State bureaucracy who is making the accusation that the school sistem does not respond to the “needs” of the learners, society, parents or local communities. That is that happens in developing countries, such as Andean countries in South America. There, the rulers and top servants of central States, are the ones who make such acusations, accusation that covertly or appenly goes against the teacher’s unions. What we have here is a twist in the “public opinion trial”, the delincuent is the accuser and the victim the accused. To understand this twist we should keep in mind that the State is the institution that control and manages the mega system called society or nation, to which educational systems belong to. So, the acusers are accusing themselves of not having the capacity to control and manage their own subsystems, the educational system. State managers should have been under trial for abandoning responsabilities, they are the delinquents. However, instead of being accused by teachers, communities and parents of the learners for not providing tochools with resources necessary for teachers to perfom their job, those State managers ?the ones who should have been in prison for no doing their job- are the ones who claim the right to accuse and not only that, the one who become re-designers of a new school system. This look like the war in Irak, the big business of destruction is being follow by the big business of reconstruction, different business under the same control, big corrupted corporations. That is the paradogical part of the issue educational reform in developing countries like Peru. And it is much more paradogical that Peru, the 2nd rich country in natural resources in the Andean zone, the one with the fastest rate of economic growth in several quinquenios during 1990-2005 in all Latin America cannot afford to finance its educational development. How to explain this fact?. We will see this later this topic. 2. Educational reforms are not designed to change the whole educational system, but to introduce piecemeal adjustments and mask-type changes to kepp the essence of the old system. That is the nature of most top-down education reforms. 3. If evaluations of educational reforms have to be performed, the evaluators should depart from those facts. The evaluators job is thus to demostrate at teoretical and methodological level the imcompetence of the designers and implementers of real and holist educational change in education. NOTE 2 “How to evaluate those incremental improvements in programs and a variety of “structural” changes called education reform?. Here we have some suggestions: “Evaluation theory development has been strongly influenced by an idealized problem-solving sequence for (a) problem identification, (b) generating and implementing alternatives to reduce symptoms, (c) evaluating the alternatives, and (d) adopting one or more of the most satisfactory (Shadish et al., 1991). Evaluation thinking differed substantially in terms of how evaluation theorists viewed such issues as purposes for evaluations, audiences for evaluation findings, roles of the evaluator, the stakeholders’ role in designing and conducting the evaluation, the extent to which goodness or value can be assigned by an external factor, the nature of knowledge produced by evaluations, and the data collection methods employed by evaluators”. J Wiley’s article [[Hugo’s note 2: PURPOSE OF EVALUATIONS]] BASIC QUESTIONS & ANSWERS REGARDING THIS RESEARCH PROJECT [[The first purpose of my evaluation will be to provide or suggest adjustments and corrections to the process of design and implementation of the designed reform from a systems view perspective. At the end of this study and according to findings I will provide some “efective” solutions to educational problems. The beneficiaries and audience of this study are ?at the beginning- the top policy makers who design the reform and those who check or control its implementation, since I will report to them some findings during the research process. The main beneficiary of this study are the communities and parents, teachers associations and stakeholders directly involved or close to the school system, to whom I will make available to final report. My role as evaluator will be strictly academic (limited to education topics of the proposed research) and neutral in political terms. I will avoid any political judgement that could hard interest groups at the top and the bottom of the spectrum. Regarding the stakeholders role in this research I will avoid their participation in the design and conducting of this evaluation, they have interest of their own that could bias the research process. Even if they are part of the donors financing this research I will make an agreement to limit our relation to information. Regarding the nature of knowledge that this study will produce; those will be the result of careful analysis and synthesis resulting from the methodology suggested by Bela Banathy. I will use his concepts, models or lenses to perform a Systems evaluation of education reforms. I will use also the original insights of theorists whose studies conform the background of Systems Theory. See in Annexes my article “Systems Theory Backgrounds”]] NOTE 3 “The critical omission of the various evaluation models produced is that they are largely silent concerning the evaluation of a system as a system. Instead, the concern is limited to the amelioration of problems possessed by existing systems or of selecting the best program or practice among competing alternatives. In short, they ignore system concepts. There is little acknowledgment that in open systems the fact that everything affects everything else, or that if one thing is altered, other conditions change in response.” Wiley’s article [[Hugo’s note 3: METHODOLOGY]] [[Does this means that “direct causal relationships seldom exist” so this type of analysis is/was irrelevant?. here we have 2 methodological points. 1st the idea that “everything affects everything else; if one thing is altered, other conditions change in response”: i guess there are key parts in a system that could produce such domino effect on other parts; however, not all of them have such impact. this means that when we analyze the structure of a school system we have to distinguish and select those parts for careful analysis if they are to be touched by a reform process. 2nd,“direct causal relationships seldom exist” . See “THE IMPACT OF SYSTEM DESIGN THINKING, 2nd papragraph, below Saying that all parts are interelated doesn’t mean that parts in a system are similar to a sack of potatos. parts in a system are structured either vertical, horizontal or a combination of both of them. so,what one part does could affect other or don’t. we can suppres some parts without affecting the continuity of the system at all, but we cannot supress some part without altering some other parts. in a school some parts of the subsystem teaching-learning ?for instance- are set in a way that a teacher of art does do not affect what the math teacher does, unless it affect time scheduled. in a factory it does, in the assembly-line units of machines, if one unit does not finish its job it affect the whole process. that is not the case of school systems. some parts are well connected, depends on one another, most of them don’t. loose coupling is the dominant feature of educational systems in the north,but direct causal relation do exist, even when multiple correlations are more relevant in some cases. at the end, the discussion of causality either lineal or multiple, is irrelevant in systems analysis. sytems approach doesn’t rely on quantitative methodology]]. “THE IMPACT OF SYSTEM DESIGN THINKING Recently, the thinking and particular formulations that have contributed to traditional evaluation theory and practice are being challenged. For example, and having substantial implications for evaluation of system designs, `naturalistic inquiry' has been proposed as a more appropriate way of thinking about and conducting research (Lincoln and Guba, 1985). The construction of a naturalist paradigm included the following axioms: (a) realities are multiple, constructed, and holistic; (b) the knower and the known are interactive and inseparable; (c) only time- and context-bound hypotheses are possible; (d) all entities are in a state of mutual simultaneous shaping, so that it is impossible to distinguish causes from effects; (e) inquiry is value-bound.” Wiley article Systems theory as naturalistic paradigm: this statement deserves some comments. Especially the 2nd paragraph with methodological insights. See hugo’s note 3, above NOTE 4 [[KEY QUESTION: WHAT IS THE PURPOSE OF THIS EVALUATION?]] SIGNIFICANCE, IMPORTANCE OR RELEVANCE? TO WHOM? “Banathy's review of approaches to `Design evaluation' reveals an increasing number of system thinkers and designers concerned with (a) evaluation as contributing to stakeholder decision-making about alternative design features (Nadler, 1981; Ackoff, 1981); (b) evaluation as helping establish `acceptability zones' for design solutions that must confront multiple perspectives (Jones, 1980); (c) evaluation as `argumentation' [[theory development?]](Rittel and Webber, 1984); (d) evaluation of design solutions through use of system criteria (Checkland and Scholes, 1990); (e) evaluation as a trade-off analysis at key decision points (Warfield, 1990); and (f) evaluation and design as complementary parts of the same process (Rowland, 1994)” [[hugo’ note 4 : more on purposes of this evaluation]] [[summarizing my purpose or objectives for this evaluation: the purpose of this evaluation study is to contribute with policy makers, stake-holders (defined as those that could or have veto power on school reforms), school actors (administrators, teachers and parents) and local communities related to the teaching-learning and developmental process of the learner in the following tasks: (a)provide corrections and adjustments to the ongoing project of education reform, or with alternative design features if the reform project has not yet approved; (b) design solutions to hot problems during the implementation process, previous evaluation of the process. this includes the suggestion to design of complementary (or subtitute) parts to the system’s aim to tackle with hot problems (adress the functional imperatives of a system, as parsons requested. (c) elaborate a report containing the analysis and synthesis of this study and diseminate it among policy makers, stakeholders, school actors and local communities. all of this with the help. (d) contribute with the theoretical development of reform evaluation and organizational theory in those areas that need more debate and reflexion, especially the issue of current community alternatives to finance and implement decentralization policies]]. [[GOOD QUOTE on WHY SYSTEMS THEORY. QUESTION WHICH PAGE IT BELONGS TO? FOLLOW BY WILEY TEXTS “Social system design seeks to understand a problem situation as a system of interconnected, interdependent, and interacting issues; and seeks to create a design as a system of interconnected, interdependent, interacting, and internally consistent solution ideas. System designers envision the entity to be designed as a whole. A systems view suggests that the essential quality of a part of a system resides in its relationships with, and contribution to, the whole” (Banathy, 1992) “Integration, then, is the sine qua non of a system design. “There are many other, equally important, characteristics of the `system design' technology; for example: (a) the importance of multiple perspectives and how these might be integrated; (b) the importance of creating an `idealized image' of the system to be that reflects the core values and needs of the system stakeholders, one that will serve as a target for design and from which, over time, a feasible and implementable operating design is extracted. “A FRAMEWORK FOR EVALUATING SYSTEM DESIGNS The main section of the paper is divided into three parts. The first part presents some general guidelines related to evaluating system designs. The second part describes a framework for thinking about evaluation as it applies to system designs. The final part discusses some limitations and complications that need to be recognized and taken into account by system design evaluators. [[First model]] “GUIDELINES FOR SYSTEM DESIGN EVALUATION “A set of guidelines are proposed below. [[OBJECTIVES]] “System design evaluations can be directed toward 1. (a) determining the appropriateness and worth of existing systems; (b) the appropriateness of new systems being designed but not yet implemented; (c) the implementation process for new systems; and (d) the `over time' appropriateness and worth of new systems. 2. The task of evaluating system designs begins with, is integrated with, and is concurrent with the systems design process itself. This means that the design process for most aspects of system design evaluations are accomplished with full involvement of system stakeholders. The design needs to be responsive to stakeholder interests. [[IN MY VIEW, THE EVALUATOR SHOULD BE INDEPENDENT, IF WANT TO PROVIDE RIGHT SOLUTIONS NOT MATTER WHAT INTERESTS ARE AT STAKE]] NOTE 5 [[hugo’s note 5: more on methodology]] “3. Social systems are in constant flux. Independent variables and, in some cases, dependent variables are vulnerable to change. Thus, capturing, understanding, and making sense of the process of change should have at least equal status to assessing effects”. (Wiley article) [[in systems theory the educ system is 1st a dependent variable and later on an independent one. the school system is 1st affected by the context and lates its outputs ?in 2nd place- affect the environment or society. so education is “in most cases” more vulnerable to changes than society is. the fact that societies can live without formal school systems but schools can not exist out of societal environments, explain the difference force and time of impacts between them. the impact of contextx on schools is stronger -especially the impact of the systemic context- than the impact of schools on society. thus, they do not have the same status, not in reality and not for the purpose of analysis and methodologycal research of reforms. on the other hand, making sense of changes in society and schools should consider that scieties are much more dynamic than schools. societies are in permanent change, schools no. the speed of changes in society cannot and will never be equated by schools. schools try to preserve, serve and reproduce yesterday social reality, are conservative in nature, while societies are revolutionary in nature, they are moving and creating today the reality for tomorrow. school systems will be always outdated, in permanent crisis because is imposible to keep up with the new demands of social change. this means that today school reform will be absolete tomorrow or in the nearest future. the school system that can survive the speed of changes of the society will not be the one without crisis ?that is a mirage- but the one that is able to respond inmediatelly to new possitive changes of that society. that is, the school that have contingency plans for tomorrows reality. to capture tomorrows reality with need to keep track and be in permanent contact with the whole society, with the external process of change in societies. this is why systems analisis became so relevant. there is not other theoretical and methodlogical framewrok that allow us to keep track of both whole systems, the society and the schools, so a quick changes in the school system or its parts can be intriduced without disrupting its cointinuitee. this is why we need to carry with us the three lenses suggested by banathy, the one to see relations between school systems and societal environments, the one to see functions & structures of the school system and the one to see both whole systems in process, the interelation and dynamic of both systems, so that we can say what need to be changed and fast inside the school system and inside the society. in sum, the process of studying education reforms start analyzing the environment, the nature and trends of their changes, that will serve the understanding of its effects on educ. then, we need to know the functional imperatives or problems to be solve in the whole structure of the school |