Thursday, March 22, 2012

Too Many Questions

On my way to CrossFit this afternoon my mind wandered back over the confused terrain of the concept map Ian and I tried to craft the other day.  Unfortunately, my thoughts started to settle into something coherent just as I got to the box so all I had time to record was a rather enigmatic text to Ian saying that I thought we were trying to squeeze too many questions into a single map.  It's many hours later and I'm not sure I can unpack everything that I was thinking at the time.  Nevertheless, I'd like to write something down before even more is lost.

In our concept map we were trying to fit theology, the tight loose distinction, environmental context, differential success and demographic variables.  Actually, we were trying to fit in even more, but these are the ones that stuck with me.  The paths of our attempted maps ended up going nowhere.  For instance, we might start by asking how theology supports, promotes or maintains tightness and looseness.  This, then, would be a causal chain, the end result of which would be two distinct kinds of groups.  These groups would, in turn, interact within their environmental context and some groups, by virtue of being either tight or loose, would be "more successful" than other groups.  This would be reflected in measurable demographic variables such as membership numbers, rates of growth or decline, amount of money sequestered, etc.

We'd draw something like this on the board and for the briefest moment it would all seem straightforward and clear.  However, these maps never held up to closer scrutiny.  We might recognize immediately that demographic variables also contribute to differential success.  For example, some groups are, on average, past reproductive age so any opportunities for bringing new bodies into the group must come from recruitment efforts.  This means that the measurable outputs are themselves contributing factors to the very phenomena we want to study.  Even worse, theology can drastically affect how a group feels about recruitment or how successful it might be in its endeavors in ways that are divorced from the way theology contributes to tightness or looseness.  If the outputs that we're proposing to use as a proxy for differential success are the result of multiple causal chains, how can we tease out the role of tightness and looseness?

And so we'd erase a new section of the chalkboard and start the whole process over again.

What occurred to me this afternoon was that at least part of the problem stems from trying to use a single concept map to answer several different questions.  Eventually, we might be able to do this, but at the moment I fear it's muddying the waters.  Our time might be better spent working out simpler maps.

For example, one of the first questions we've asked ourselves is, what are the environmental conditions that favor either group tightness or looseness?  Conceptually, this is a relatively clean question and one we're already addressing with our modeling project.  If the Templeton grant comes in and we are able to recruit an historian, we could bring a host of historical information to bear on this question as well.  In the meantime, we can survey or otherwise census local congregations for where they fall on the continuum between tight and loose, get their demographic information for some number of years so we know whether they're growing, declining or holding steady and, finally, look to see how individual members are faring.  In other words, we have a number of potential sources of data that can tell us how different groups are doing, or are likely to do, in different environmental contexts.  I haven't tried to sketch this out as a concept map, but I'm willing to bet it would be a lot cleaner than anything we left on the lab board so far.

But where does theology come in?  Well, it potentially comes in at a number of different places and we'd have to take up its role on a case by case basis.  Each case, though, would pose a different set of questions with potentially a different set of maps.  The one I'm working on right now is the role of theology in supporting, promoting and maintaining tightness and looseness.  Essentially, this is an exploration of proximate mechanisms and, as such, should probably be held apart from the question of differential success, which speaks more to ultimate causation.  Trying to mix the two, at least at this point, seems quixotic.  Although they are intimately related questions, they are nevertheless different and the metrics we use to assess one set of questions don't relate in a direct way to the metrics we use to assess the others.

Simplify!

4 comments:

  1. We really need a virtual whiteboard for this blog.

    At any rate, I take your point about tackling specific and relatively isolated questions, with an eye towards stitching it coherently together at a later date. The problem is that, in some sense, a lot of this work has already been done by social scientists. So what we bring to the table is a set of novel terms (tight/loose) that are essentially synonymous with existing (strict/lax), and a theoretical framework upon which it all can hang convincingly. And it's the hanging that's causing the headaches...

    ... ok, that's the fourth paragraph I've written here and rather than send it the way of the previous three (deleted) I'll leave it and start reading my book on structural equation modeling.

    ReplyDelete
  2. Yes we do!

    I realized as I was writing this post that structural equations might require us to put more variables on the board at once than I would otherwise be comfortable with. I look forward to learning more about this--through you, of course. I have absolutely zero desire to put that particularly learning curve in my path at the moment!

    ReplyDelete
  3. Not quite. We can start with distinct sections and work out those piecemeal.

    The best bit about SEM is that you have observed variables (what you measured through surveys etc) and latent variables (estimates of factors that observed variables cluster around) and can look at the interactions between any combo of the two (and then some for advanced users). So we could have different indicators of tight/loose loading onto our latent diversity tolerance variable. Kind of cool. It even handles multi-level stuff. However, I get the distinct impression that data prep is an absolute bitch, along with interpretation ;)

    ReplyDelete
  4. And can you do all of this in SPSS? On the one hand, it all sounds very promising and exciting. On the other hand, it strikes me as being a bit like semantic network analysis or factor analysis--more art than science.

    ReplyDelete