MENU

   Publications Home

   C4I Home
   Center Overview

   C4I People

   Objectives
   Programs
   Funding & Support
   Industry Partners

   Publications
   Recordings

   C4I Events
   News

   Internet Conference

C4I Publication Abstracts
Command Support

C3I-2001

Examining the Effect of Causal Focus
on the Option Generation Process:
An Experiment Using Protocol Analysis

Authors: Leonard Adelman, James Gualtieri, and Suzanne Stanford

An experiment using protocol analysis found that information providing different causal foci resulted in (1) the generation and selection of different types of causal hypotheses and (2) the generation and selection of different types of options. These "framing" effects were found for an ill-structured problem with multiple causes and a wide range of viable options. These findings extend previous research on problem solving and decision making under uncertainty which has used different types of tasks and not focused on option generation per se. In addition, the findings extend previous research on option generation, which has not examined the effect of causal focus on option generation or selection. Additional analyses suggest that the option generation process can be represented as a cycle of option generation, clarification, and evaluation against certain criteria specified by the frame, with periodic breaks for considering data and, to a lesser extent, the causes of it. The selected options tended to be those that had been evaluated, clarified, and generated again (i.e. regenerated) more often during the course of the option generation process. To use an analogy, the causal focus pointed the option generation rifle; option evaluation, clarification, and generation pulled the trigger.

Organizational Behavior and Human Decision Processes, Vol. 61, No. 1, pp. 54-66, January 1995.

C3I-2002

Examining the Effect of Information Order
on Expert Judgment

Authors: Leonard Adelman, Martin A. Tolcott, and Terry A. Bresnick

An experiment with trained Army air defense personnel performing a paper-and-pencil representation of their substantive task was performed to test the predictions of the Hogarth-Einhorn model (1992) for belief updating. For a simple task with a short series of information, the model predicted a significant information order by response mode interaction. The experiment supported the model's predictions. When information was presented sequentially and a probability estimate was obtained after each piece of information, the order in which the same information was presented significantly affected the final mean probability estimates. In contrast, when all the information was presented at once and a probability estimate was obtained at that time, information order had no effect. There were, however, large individual differences. Moreover, order and response mode cumulatively accounted for less than 10 percent of the total variation in the participants' probability estimates. These findings, and their implications, are discussed.

Organizational Behavior and Human Decision Processes, 56, pp. 348-369, 1993.

C3I-2003

Examining the Effect of Information Sequence on
Patriot Air Defense Officers' Judgments

Authors:Leonard Adelman and Terry Bresnick

This paper describes a recent experiment conducted with Patriot air defense officers and employing the Patriot air defense simulators at Ft. Bliss, Texas. The experiment found that, under certain conditions, the participants made different identification judgments and took different engagement actions depending on the sequence in which the same information was presented to them. This finding was consistent with the theoretical predictions of the Hogarth-Einhorn belief updating model. However, there were large individual differences, and experience mitigated the effect. These findings replicated previous research by the authors using a paper-and-pencil instrument. Limitations in the current experiment are discussed, as are directions for future research.

Organizational Behavior and Human Decision Processes, Vol. 53, pp. 204-228, 1992.

C3I-2004

Experiments, Quasi-Experiments, and Case Studies:
A Review of Empirical Methods for Evaluating Decision Support Systems

Author: Leonard Adelman

Developers of decision support systems (DSS) often fail to present empirical data supporting the claimed merits of their systems. Discussions with developers indicate that they often do not consider or know how to perform the required empirical evaluations. That problem is addressed by reviewing the issues inherent in using experiments, quasi-experiments, and case studies to evaluate DSSs. The paper is meant to be a tutorial for graduate students and engineering professionals developing DSSs. The discussion revolves around the issues of reliability and four types of validity: internal, construct, statistical conclusion, and external. The DSSs are focused upon, including expert systems, but the discussion is appropriate for most interactive systems.

IEEE Transactions on Systems, Man, and Cybernetics, Vol. 21, No. 2, March/April 1991.

C3I-2005

Using the Multitrait-Multimethod Matrix
to Evaluate Knowledge-Based Systems

Authors: Leonard Adelman and Sharon L. Riedel

This paper illustrates how the multitrait-multimethod matrix was used to assess the technical adequacy of a Knowledge-Based System (KBS) that used Multi-Attribute Utility Assessment parameters to represent knowledge. The traits were three subject matter experts' (SMEs) knowledge of critical tradeoff judgments, represented as relative importance weights. The KBS and the Simple Multi-Attribute Rating Technique were the weighting methods used. We found that, on the average, the convergent validity coefficient measuring the between-method agreement for each SME was significantly greater than both types of discriminant validity coefficients measuring between-SME agreement using either the same or different methods. This finding permitted us to conclude that even though there were obvious differences in the weights generated by the two methods, the methods still generated weights that appear to reflect the SMEs' tradeoff judgments. The concluding section of the paper discusses how the multitrait- multimethod matrix can be applied more generally to assess the adequacy of KBSs in difficult but common evaluation settings like the one we faced; that is, where there is only one problem scenario, SMEs who disagree, and no accuracy measures.

Invited submission to a special issue entitled "Evaluation of Knowledge-Based Systems" for Communication and Cognition - Artificial Intelligence, Paper written December 1993.

C3I-2006

A Multifaceted Approach to Evaluating Expert Systems

Authors: Leonard Adelman, James Gualtieri, and Sharon L. Riedel

A multifaceted approach to evaluating expert systems is overviewed. This approach has three facets: a technical facet for "looking inside the black box," an empirical facet for assessing the system's impact on performance; and a subjective facet for obtaining users' judgments about the system. Such an approach is required to test the system against the different types of criteria of interest to sponsors and users and is consistent with evolving life-cycle paradigms. Moreover, such an approach leads to the application of different evaluation methods to answer different types of evaluation questions. Different evaluation methods for each facet are overviewed.

Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 8, pp. 289-306, 1994.

C3I-2007

Real-Time Expert System Interfaces, Cognitive Processes,
and Task Performance: An Empirical Assessment

Authors: Leonard Adelman, Marvin S. Cohen, Terry A. Bresnick, James O. Chinnis, Jr., and Kathryn B. Laskey

In this experiment we investigated the effect of different real-time expert system interfaces on operators' cognitive processes and performance. The results supported the principle that a real-time expert system's interface should focus operators' attention on where it is required most. However, following this principle resulted in unanticipated consequences. In particular, it led to inferior performance for less critical, yet important cases requiring operators' attention. For such cases operators performed better with an interface that let them select where they wanted to focus their attention. Having a rule generation capability improved performance with all interfaces but the improvement was less than hypothesized. In all cases, performance with different interfaces and a rule generation capability was explained by the effect of the interfaces on cognitive process measures.

Human Factors, 35(2), pp. 243-261, 1993.

C3I-2008

Composing and Constructing Value Focused Influence Diagrams:
A Specification for Decision Model Formulation

Authors: Daniel T. Maxwell and Dennis M. Buede

Over the past three decades, significant improvements in the computer and computational sciences have enabled automated support for increasingly complex decision situations. One example of this progress is the influence diagram. While these improvements are valuable to the field of decision analysis, current graphical methods do not specify a cohesive and comprehensive process for identifying and structuring the interactions of options, values, and uncertainties during the initial development of a decision model. We introduce the first formally specified, integrated process for structuring complex decision models, and call the resulting model a Value Focused Influence Diagram. Influence diagram vocabulary is extended to include a formal definition of special case nodes that articulate explicitly a decision maker's fundamental objectives. Proofs demonstrate the process' exhaustiveness, soundness, and consistency with current computational conventions of influence diagrams. The assessment process decomposes the decision situation and utilizes a collection of structured queries to identify relevant issues of fact and value. Specifically, Decision Directed Similarity Networks are introduced as an improved method for refining the value and decision space, and Bi-Polar Assessment Graphs are introduced to identify a set of relevant uncertainties that explain extreme values of attributes in submodels. After the structured queries provide the requisite information for the submodels, the construction stage of the process organizes the information collected during composition into a single network that preserves the integrity of the probabilistic model implied by each of the submodels. Additionally, the value focused process increases the emphasis on a decision maker's values, and helps to correct for common judgmental biases. Our decision structuring process defines and reduces the assessment burden on decision makers and decision analysts.

Submitted to Operations Research , December 1994.

C3I-2009

Convergence in Problem Solving:
A Prelude to Quantitative Analysis

Authors: Dennis M. Buede and David O. Ferrell

This paper describes a set of procedures that could be successfully used to restructure a mental model of diverse connected constructs into a compact model suitable for quantitatively examining the key options available to the decision maker. These procedures can be applied in an interactive manner to a single decision maker or a group of decision makers. The input to the convergence process is a cognitive map. A cognitive map is a signed, directed graph that links important concepts in the decision problem on the basis of probabilistic and informational dependence. It is assumed that the nodes in the graph have been labeled as decisions, uncertainties, or values. The convergence procedures first structure the value and decision ends of the cognitive map so that each end properly reflects the value structure and a sample of varied and important options open to the decision maker. Sets of uncertain nodes are then identified as good candidates for aggregation or deletion. This identification is achieved by graph operations based upon specifically defined graphical structures and selected qualitative reasoning techniques.

IEEE Transactions on Systems, Man, and Cybernetics, Vol. 23, No. 3, May/June 1993.

C3I-2010

Rank Disagreement:
A Comparison of Multi-Criteria Methodologies

Authors: Dennis M. Buede and Daniel T. Maxwell

A number of multi-criteria decision support techniques have emerged in recent years that use varying computational approaches to arrive at the most desirable solution and thereby "recommend" a course of action. Decision makers who use the results of this analytic work should be assured that the computational schemes used by their supporting analysis or decision support software produce the appropriate solutions. We conducted a series of simulation experiments that compared the top-ranked options resulting from the computational algorithms that support Multi-Attribute Value Theory (MAVT) and three methods that are reported in the literature that allow rank reversals, the change in rank order of two options when an unrelated option is added or deleted from the analysis: the Analytical Hierarchy Process (AHP), Percentaging, and the Technique for Order Preference by Similarity or Ideal Solution (TOPSIS). We also included a fuzzy algorithm proposed by Yager to gauge its consistency with the other algorithms, even though it is not subject to rank reversals. These experiments demonstrated that the MAVT and AHP techniques, when provided with the same decision outcome data, very often identify the same alternative as "best". The other techniques are noticeably less consistent with MAVT, the fuzzy algorithm being the least consistent. The situations under which the most frequent and significant differences occurred were dependent upon the method.

The results of our experiments indicate that other issues (e.g. the processes used for problem structuring and the elicitation of value weights) are likely to be of greater significance to problem outcome (based on our experience) than the choice between the computational algorithms of MAVT and AHP. The results cause us to be concerned about the use of the other methods.

Journal of Multi-Criteria Decision Analysis, Vol. 4, pp. 1-21, 1995.

C3I-2011

Superior Design Features of Decision Analytic Software

Author: Dennis M. Buede

This paper defines the critical capabilities of decision support software and highlights the superior analytic and user interface features of the 32 commercially available decision analytic software packages. The packages provided a mix of capabilities related to problem structuring, value matrices, multi- attribute utility analysis, decision trees, and influence diagrams. First, we define the process of decision making in organizations. Next, we summarize the 32 software packages that were reviewed. Then we present and discuss the evaluation structure and superior design features.

Computers Operations Research, Vol. 19, No. 1, pp. 43-57, 1992.

C3I-2012

Providing an Analytic Structure of Key System Design Choices

Authors: Dennis M. Buede and Robert W. Choisser

The design of a system, especially a system architecture, requires the balancing of multiple performance parameters with the system cost and other implementation issues. Multi-attribute utility theory provides a structured, coherent framework for conducting an analysis in which system designs are evaluated on multiple performance parameters. This paper describes the application of multi-attribute utility analysis to the design of the Worldwide Digital Signal Systems Architecture (WWDSA), a telecommunications system of the United States' Defense Communications Agency (now the Defense Information Systems Agency). The advantages of using multi-attribute utility theory during the system design are highlighted. In addition, we discuss several key analytical issues that led to this application being a success in the eyes of the decision makers.

Journal of Multi-Criteria Decision Analysis, Vol. 1, pp. 17-27, 1992.

C3I-2013

Risk/Decision Analysis for Trade Studies

Author: Dennis M. Buede

This paper relates the top-down, iterative design process of systems engineering to key elements of decision analysis to show how decision analysis concepts can be used to enhance decision making by systems engineers. We begin by reviewing the systems engineering process: requirements, operational concepts, system boundaries, and functional, physical and operation architectures. We then discuss such requirements as objectives and constraints. Next we introduce the decision analysis concepts of the fundamental objectives and the fundamental objectives hierarchy. These ideas are related to the traditional concepts of measures of effectiveness (MOEs) and measures of performance (MOPs). We then describe influence diagrams and show how the choice of operational architectures and the associated requirements allocation can be addressed with influence diagrams. We discuss how the fundamental objectives and objective hierarchy change as the iterative design process proceeds to greater and greater detail. The analogy is drawn between physical interfaces between components of the design and the requirements allocation decision interface between layers of design detail.

Proceedings of the National Council of Systems Engineers Symposium, 1995.

C3I-2014

Probabilistic Reasoning for Assessment of Enemy Intentions

Authors: Kathryn Blackmond Laskey, Suzanne Mahoney, and Col. Bernard Stibio

A military commander must reason about the intentions of an opponent actively attempting to deceive him. Threat assessment is the process by which intelligence analysts put together information from various sources to reason about the likely actions and intentions of the enemy. This paper considers the use of probability to represent and reason about uncertainty regarding enemy intentions. Issues in knowledge representation, inference network construction, control of reasoning, and interpretation of outputs are discussed.

Submitted to IEEE Transactions on Systems, Man, and Cybernetics.

C3I-2015

Bayesian Learning of Markov Field Models

Authors: Kathryn Blackmond Laskey and Laura Martignon

Markov fields have been applied as models of both natural and artificial neural networks. The graph of a Markov field encodes connections between neurons and local parameters encode the strength of neural interactions. A Bayesian learning algorithm is presented for inferring structure and parameters from a database of independent and identically distributed cases from the distribution. We present a conjugate family of prior distributions for model parameters conditional on connectivity structure. The Bayesian Information Criterion is used to approximate the posterior probabilities for structures. The method is illustrated on a problem of learning the connectivity structure of a set of neurons from data on neuron firings. Application to learning concepts from data on coincident activation of features is discussed.

Submitted to Neural Networks.

C3I-2016

Model Uncertainty:
Theory and Practical Implications

Authors: Kathryn Blackmond Laskey and Vicki M. Bier

A model is a representation of a system that can be used to answer questions about the system. In many situations in which models are used, there exists no set of universally accepted modeling assumptions. The term "model uncertainty" commonly refers to uncertainty about a model's structure, as distinguished from uncertainty about parameters. This paper presents alternative formal approaches to treating model uncertainty, discusses methods for using data to reduce model uncertainty, presents approaches for diagnosing inadequate models, and discusses the appropriate use of models that are subject to model uncertainty.

To appear in IEEE Transactions in Systems, Man, and Cybernetics.

C3I-2017

Sensitivity Analysis for Probability Assessments
in Bayesian Networks

Author: Kathryn Blackmond Laskey

When eliciting a probability model from experts, knowledge engineers may compare the results of the model with expert judgment on test scenarios, then adjust model parameters to bring the behavior of the model more in line with the expert's intuition. This paper presents a methodology for analytic computation of sensitivity values in Bayesian network models. Sensitivity values are partial derivatives of output probabilities with respect to parameters being varied in the sensitivity analysis. They measure the impact of small changes in a network parameter on a target probability value or distribution. Sensitivity values can be used to focus knowledge elicitation effort on those parameters having the most impact on outputs of concern. Analytic sensitivity values are computed for an example and compared to sensitivity analysis by direct variation of parameters.

To appear in IEEE Transactions in Systems, Man, and Cybernetics.

C3I-2018

Meta Reasoning and the Problem of Small Worlds

Authors: Kathryn Blackmond Laskey and Paul E. Lehner

Practical decision theoretic reasoning requires the construction of local, problem-specific models in which attention is confined to a restricted universe of propositions. Such a restricted universe is called a small world. Managing the construction and revision of small world models can itself be viewed as a meta-level decision problem. This paper presents a theoretical framework for analyzing the management of small world models and presents results of a simulation study of meta-reasoning policies for model diagnosis.

IEEE Transactions on Systems, Man, and Cybernetics, 24(11), pp. 1643-1652, 1994.

C3I-2019

Bounded Rationality and Search over Small World Models

Author: Kathryn Blackmond Laskey

The ideal Bayesian agent reasons from a global probability model. Real agents must use simplified models which they know to be adequate only in restricted circumstances. Very little formal theory has been developed to help fallibly rational agents manage the process of constructing and revising small world models. The goal of this paper is to present a theoretical framework for analyzing model management approaches and to illustrate the approach by analyzing a particular class of model management strategies. For a probability forecasting problem, a search process over small world models is analyzed as an approximation to a larger-world model which the agent cannot explicitly enumerate or compute. Conditions are given under which the sequence of small-world forecasts converges to those produced by the larger world model.

International Journal of Approximate Reasoning, 11(4), pp. 361-384, 1994.

C3I-2020

Probability and Plan Evaluation in Knowledge-Based Planning

Authors: Paul E. Lehner and Christopher Elsaesser

Research in decision theoretic planning has concentrated on uncomplicated plans. Planning systems that generate temporal belief networks have been limited to linear plans, a single abstraction level, and restricted planning horizons. Decision theoretic planners that do not construct temporal belief networks have been limited to action models with single input and output situations (i.e. state transitions), do not allow temporal references, and cannot perform backward projection during plan monitoring. Such planners are inadequate to evaluate the complex plans that can routinely be generated by many knowledge-based planners. In this paper we show how to construct temporal belief networks to evaluate plans far more complex than those addressed by current decision theoretic planners. Our mechanism addresses plans that include multiple contingencies, varying levels of abstraction, multiple and variable duration actions which overlap temporally, and action models with effects that occur during action execution and are contingent on other events that occur during execution.

C3I-2021

Sensor Planning for Elusive Targets

Authors: Scott Musman, Paul E. Lehner, and Chris Elsaesser

Problems such as searching for units in a military battlefield, attempting to detect and interdict illegal drugs before they are brought across an international border, skip tracing, and searching for alleged criminals in law enforcement all involve generating plans to apply severely constrained information collection resources to search for intelligent agents that are actively avoiding discovery. An automated approach for generating plans to search for "elusive agents" is presented, along with an instantiation of this approach to search for mobile ground moving targets. The approach (1) uses automated mobility and terrain analysis to enumerate a suite of alternative plans for the target agents, (2) uses sensor models to identify observation windows (i.e. space/time regions where the target agents may be detected if they are executing one of the enumerated plans), and (3) generates a search plan for multiple mobile sensors by heuristically searching through alternative subsets of the observation windows. Each search plan, defined as a temporally-ordered subset of the observation windows, is evaluated by exercising an automatically constructed Bayesian network that summarizes the results of the terrain, mobility, and sensor coverage analysis. An empirical evaluation of this system was performed with results supporting this utility.

Submitted to Mathematical and Computer Modeling.