[21]                             home                            [23]

 

 

 

 

 

 

The Harvest of Social Discourse

 

 

 

4/8/2004 2:29 PM

 

Background

Relevance

 

Background

 

First some background.  Several government projects have involved the harvesting of social discourse as represented by text posted on the Internet.  The capability has never existed in the past.  Public opinion polls and standardized tests are a shadow of what is possible with the real time analysis of human language posted in the open on the millions of web sites.  Increasingly, personal web logs, blogs, springing up.  Blogs, e-forums and traditional news oriented web sites provides the source materials to a harvest of human text from which a detailed understanding of the threads in shared thought can be obtained.  One only needs the proper technology. 

 

The J-39 project originated in the George W. Bush administration at a National Security Council meeting (private communication).  This project involved the Open Source monitoring of many public web sites where Islamic society talked about political issues.  Some of the harvesting was of English only text and some was oriented towards harvesting Arabic.  The harvesting was designed to reveal the variation in opinion from one country to another country, and to be able to identify specific threads of discourse. 

 

I came to know the computer architecture for the harvesting process and the technical means for the representation of social opinion in my role as Senior Scientist at one of the contractors whose work included a major role in the J-39 system. 

 

My evaluation was informed by a detailed understanding of the research literature in automation of back of the book subject indexing, text understanding, machine translation, human cognitive processes, neural and functional architectures related to selective attention, memory and anticipation, general systems theory related to memetic expression, social network theory, social science, and related topics. 

 

In my professional opinion, the architecture and technical means was seriously flawed.  Some of the flaws could be easily corrected, but where not.  Many of the dysfunctions that have come to characterize large government agencies were present.  However, a deeper problem persists related to how well people really understood the technical requirements necessary to understanding social discourse in 40 – 50 countries.  A number of paradoxes make this understanding elusive.  Text understanding by mature humans is easy and immediate.  So it is quite natural to infer that computer technology can do what is easy for humans to do.  In fact this inference is encouraged by marketing literature.

 

Some of the J-39 system flaws required adjustments to theoretical considerations.  My work, over a period of at least a decade, had more and more focused on the delineation of these theoretical considerations.  It is understandable that most experts do not follow the arguments I have been making.  The issues are far from those typically addressed and there is more than one important issue.

 

The measurement problem is perhaps the most essential, and the one ignored the most.  In human conversations the uptake of experience is barely noticed, if at all.

 

At the core of my 2001 – 2202 evaluation was the wide adoption, by contractor and clients, of a seven step “Actionable Intelligence Process Model”.  The seven-step model was missing the first two steps in a new model that I developed and made known (Figure 1). 

 

 

Figure 1: The nine step model : Prueitt (2002)

 

The seven-step AIPM started with “detecting facts and events”.  The missing two steps created a critical weakness in the standard model.  The same weakness is seen in academic research in the area of database schema, artificial intelligence, semantic web standards, machine-learning, and computer simulation.  But the theoretical considerations related to the missing steps have extensive supporting literatures.  In some academic disciplines the issue is codified as “the measurement problem”.  It can be stated as follows:

 

How does a living and behaving animal perceive the world in real time and orient behavior within the real time environment experienced? 

 

The measurement problem is a hard problem. 

 

But my work, and discussions with scholars and technologists, had developed some simple and surprising results.  This new effort extended work I had done on categorical abstractions and event chemistry in 2000 and 2001 in the context of measuring Internet transactional types to detect the nature and origins of cyber intrusions. 

 

A new technology is being shown to radically simplify the development of a weathermap-type eventmap of real time event occurrences using what became called SLIP (Shallow Link analysis, Iterated scatter gather, and Parcelation). 

 

 

Figure 2: One of several overall architectures (2002) for using SLIP

within a sense making framework

 

Analysts discussed with me, and with several others, their hope that some improvements might be made in how the harvesting and concept representation system worked.  I discovered several innovations and put together a framework that would work better (Figure 3).  As I begin to push on the issues to try to implement small changes in the existing system, I come up against the same set of social challenges that I had seen before.  Most of the decision-making is based on the financial needs of the private consulting firms.  Because the technical issues are so easy to ignore and so very difficult, given the approaches made in computer science, the primary focus has become financial needs and marketing.

 

In the absence of clear and definitive science to address the technical flaws, very little can be done to end a set of confusions.  However, the importance of the problem carries with it great economic weight.

 

 

Figure 3: A detailed technical diagram for technologies supporting

the high fidelity measurement of social discourse

 

My efforts as senior scientist come to an end when the Prime contractor alerted my employer of criticism that I had made about the system.  After resigning, I received a three-month contract to complete some of the architectural work.  After these three months, I worked on patent disclosures and technology analysis for several firms, and extended the foundational work on SLIP to what is now called Ontology referential bases (Orbs).

 

Relevance

 

So what does this have to do with branding philosophy and technology supporting a harvest of text from social discourse? 

 

The replacement of advertising with brand marketing has both positive and negative consequences to society.  The positive aspects have to do with understanding what humans want and need so that products and services are made available.  

 

The negative aspects would develop if companies used the web harvesting products to suppress and smother concerns that consumers might begin to express.  One could imagine having company contractors in chat rooms or contractors posting within blog systems in order to express blocking opinion in any case where a concern was expressed.  The contractor might say that they have the same product and that everything about the product is fine, and then suggest that the person making the original comment was know to be working for a competing firm.  One already sees the practice of companies employing individuals to be in chat rooms and talk up the company’s product.

 

There are many caveats to consider, if one is going to harvest conversations and process that harvest into high fidelity measures of social discourse.  E-forums do not allow balanced presentations of viewpoints.  Those who have extensive experience with e-forums know that these who will exhibit certain behaviors will dominate the discussion. 

 

A broader concern should be discussed.  I, and others, have designed technology for examining real time thematic structure of social discourse.  When this technology is fully developed it will have a capacity for knowledge management of a type really not seen in today’s political, educational, cultural, and economic practices.  But the technology is not fully developed and is lacking in some essential details.

 

If the technology is not developed completely, the risk is that companies will turn away from the positive promise and use the technology in a narrow and exploitive fashion.  For example the emphasis on identifying rumors about a company product could turn into a means to identify and threaten anyone who would express concerns of a specific type. 

 

I need to be clear here.  The question is not about intent.  My question is about a new industry attempting to replace the advertising industry with an understanding of social discourse.  This could be very positive.  However, if the early attempts at establishing a thematic analysis of social discourse is done poorly, then the advertising industry may marginalize the more positive aspects and diminish the degree to which the actual opinion of everyday people matter.