• Introduction
  • Speech Act Model
  • Speech Act Process Model
  • Speech Act Process Flow Model
• Evaluation
  • Phase I
  • Phase II
• Future Work
• Publications
• Mailing List
• Contact

---

Introduction

---

What? The sMail (Semantic Mail) Conceptual Framework applies speech act theory to email communication processes, in order to give a formal structure and semantics to ad-hoc workflows which are characteristic of email communication. The complete lack of visibility of these workflows leads to Email Overload - the user is expect to deal with too much information in too many different ways. As a direct result of the mismanagement of these workflows, the productivity of people that rely on email communication suffers greatly. If this knowledge can be elicited, it becomes possible for the machines to understand and support the user with managing and keeping track of the ongoing workflows. If this sounds like an impossible feat, read on to see how the sMail approach promises to solve this problem.

How? The content of an email message content can be summarized into a number of pre-defined speech acts. These speech acts are instances of the sMail Speech Act Model - a speech act conceptualization specific to the electronic communication domain with a particular emphasis on (but not only) email. An exchanged speech act amounts to a speech act Process. Such an exchanged speech act implies an effect (to varying degrees) on both the sender and receiver of the speech act. The sMail Speech Act Process Model outlines a number of different effects and applies them to pre-defined instances of the speech act model. Thus, every single speech act elicited from within a new email message can be seen as the start of a separate workflow. The effects implied in a speech act process are non-deterministic - which means that they do not necessarily have to occur. However, they predict what is expected to occur when a speech act is sent or received. On the sender side, this can include managing new personal information - i.e. adding an email generated task to the user's task list. On the receiving side, apart from managing new personal information it may also require a reply email. This reply email can include speech acts that are in reply to the speech acts in the original mail - thus signifying a later snapshot of the evolving workflow; or new independent speech acts - signifying a separate new workflow. Therefore, any email in any thread can contain any number of speech acts - all of which represent a separate workflow in different stages of lifetime. The sMail Speech Act Process Flow Model outlines the dynamic behaviour of these workflows within the email communication process over time, given the expected effects of an exchanged speech act outlined in the previous model.

If all this knowledge about the numerous ad-hoc email workflows can be captured and exposed to machines, the email communicators can be supported with email management in a number of different ways. The framework includes the following conceptual models:

1. Speech Act Model - Defines concepts which best represent various implicit speech acts associated with explicit text segments.
2. Speech Act Process Model - Outlines the expected effect of individual speech acts on the communication participants.
3. Speech Act Process Flow Model - Outlines the dynamic behaviour of ad-hoc email communication process workflows over time.

The knowledge in the first two models was captured within the sMail Ontology - in this way exposing it to machines. The sMail Ontology uses concepts from NEPOMUK Social Semantic Desktop Ontologies, thus rooting the knowledge within existing knowledge representing items on the user's desktop. These items, or representations of people, files, appointments, tasks, projects etc, are frequently the artefacts of email communication workflows. Therefore, grounding the sMail ontology within NEPOMUK means that the knowledge harvested from email workflows can also be exposed and linked to knowledge on the user's desktop. This gives a broader scope to our Semantic Email while benefitting the social side of NEPOMUK at the same time.

The first model has already been evaluated in Phase I below. Evaluation of the second and third models will be carried out indirectly via the use of Semanta. We will now provide more information about these models individually.

Speech Act Model

This model was based on a succession of previous work in the area, most notably by Carvalho&Cohen [1]. We define the sMail Speech Act as the triple (a,o,s); where a denotes an action, o the object of the action and s the subject of the action.

The Speech Act Model contains instances for these speech act parameters. Actions (Request, Assign, Suggest, Deliver, Abort, Propose and Decline) can have varying Roles, e.g. Initiative for actions used to initiatite discourse and Continuative otherwise (see Fig. 1). Actions may serve particular roles in different situations (e.g. Deliver can be Responsive as a response to a request or Informative otherwise).

The Object represents instances of Nouns (see Fig. 2), the nouns being Data - representing something which can be represented within email (Information, Resource, Feedback); and Activities - representing external actions occurring outside of email (Task, Event).

The Subject is applicable to speech acts having an activity as their object. It states who is involved in that activity - the Sender (e.g. “Can I attend?”), the Recipient (e.g. “You have to write the document”) or Both (e.g. “Let's meet tomorrow”).

Speech Act Process Model

The Email Speech Act Process Model considers each speech act as a separate process. Speech act theory highlights the three forces of utterances - the Locutionary (literal meaning); the Illocutionary (social function the speaker is performing); and the Perlocutionary (the result or effect on the hearer in the given context). The sMail Speech Act Process Model models the latter two forces for all combinations of speech acts given in our model. In essence, it outlines the expected reaction from both initiator and participant of a speech act, on sending it and on receiving it respectively. It assigns the Initiator Expected Action [IEA] and the Participant Expected Reaction [PER] to each speech act combination, and is applied to the Speech Act Model as (v,o,s) [IEA] » [PER]. The IEA refers to the status or action of the speaker, or in this case, the initiator on sending a speech act (Expect, None). The PER refers to the reaction expected from the hearer, or in this case the participant upon receiving and acknowledging a speech act (Reply, Perform, None). Expect denotes that the communicator is expecting further communication. Reply denotes that the communicator is expected to further the communication, whereas Perform denotes that on sending or receiving a speech act the communicator is expected to perform an external action as a direct result of the speech act (e.g. attend an event/perform a task). Table 1 shows all combinations (Action, Noun - generalization of the Object, and Subject) of speech acts in the sMail speech act model with the associated IEA and PER.

Speech Act Process Flow Model

Detailed information about the sMail Speech Act Process Flow Model (also referred to as the Email Speech Act Workflow [ESAW]) is documented in our ESWC2008 publication. In brief, the workflow model investigates possible sequences of speech acts and other actions within email discourse or threads, given the speech act process model. Thus, it can be considered as a formal representation of the ad-hoc workflows taking place within email communication, e.g. Meeting Scheduling, Task Delegation, Data Request, Event Announcement etc. In order to get a visual idea of the workflow we have provided an animated walkthrough at the end of this section.

The ESAW models these workflows as sequences of speech-acts occuring in email conversations. It attempts to generalise all the possible scenario given a speech act is exchanged in an email (e.g. a Request-Information speech act). In general there are two possible scenario's occuring after the recipient acknowledges an incoming speech act:

1. The recipient needs simply acknowledge the speech act, no further action required (e.g. the recipient acknowledges some delivered information).
2. Upon acknowledgment of a speech act, the recipient provides a reply to the speech act, where the reply itself is annotated as another speech act (e.g. providing requested information, annotated as a Deliver-Information)
3. Upon acknowledgment of a speech act, the recipient is expected to do something (e.g. the recipient accepted a task, or acknowledged an event to be attended) as a consequence.

These scenarios may happen separately or together. For example, when the recipient approves a request for a meeting: a) the recipient sends notification of their approval back to the sender b) the recipient is expected to attend a meeting as a consequence.
Given the ad-hoc nature of these workflows (e.g., there is no way one can be sure of how a recipient will react to a Request in an incoming email), the ESAW allows for any kind of reaction that can be represented by our Email Speech Act Model. Thus there can be situations where a new instance of the ESAW is initiated within another ESAW. For example, upon receiving a request for a meeting (which constitutes the original workflow), the recipient might ask and wait for more information regarding the meeting, before going on to approve or disapprove the meeting. This 'ask and wait' process is in fact a secondary workflow within the original workflow.

The ESAW is extendable, and new processes can be easily modelled and integrated. Additionaly, new speech acts and their behaviour can be represented in the Speech Act Model and the Speech Act Process Models. These could also be easily represented within an extended ESAW.

The current ESAW is presented in Figure 4. The workflow figure is assisted by a Legend depicting the workflow patterns used and the corresponding UML 2.0 notations. Some abstractions have been performed to keep the workflow as simple as possible. Nevertheless, the workflow remains too heavy to be digested. Thus, we have provided a simple animated walkthrough below.

The following is a simple walkthrough for a 'Meeting Scheduling' scenario. In the video one can trace the executed path in the ESAW for this given workflow.

---

Evaluation

---

The evaluation of sMail is split in two phases. Phase I deals with the evaluation of the sMail Speech Act Model, whereas Phase II deals with the evaluation of the other models. In general, and especially following each evaluation, the sMail framework is open for improvement. Therefore the models discussed in the evaluation phases may not reflect the models presented in the previous section - which always shows the most up to date models.

Phase I

(This evaluation is documented in our LREC2008 publication)
In order to perform our evaluation we required an appropriate statistical methodology that serves two purposes:

1. To measure the model’s goodness of fit when applied to real data.
2. To compare this measure with that for the Carvalho&Cohen model [1].

This measure can be obtained by calculating the inter-annotator agreement between human annotators annotating segments of a corpus of emails with one or more speech acts. Of the available methodologies we chose the Kappa statistic, which may be computed as:

where κ is the Kappa coefficient, P(A) is the total agreement, and P(E) is the percentage of agreement of raters which occurs by chance alone. The value of Kappa ranges from -1 to +1, 1 being complete agreement. Despite that the same statistic was used to evaluate the Carvalho&Cohen model (henceforth referred to as the CC model), the statistic they obtained in their work could not be directly compared to the statistic which we were measuring, given that the selected email corpus would have been different. Therefore we instructed two annotators to carry out two separate annotation experiments using the same corpus to calculate κ for both models. The corpus for the experiment was comprised of a random selection of 50 email threads from the Enron corpus which discussed social, academic, and corporate issues.

The annotation tasks for the sMail speech act model required annotating multiple text segments within an email and not the email itself. These segments were not pre-agreed upon by the annotators. Where necessary, it was agreed to assign more than one speech act to a single text segment. The speech act combinations for the sMail and the C&C models were used as categories for the κ statistic for the two experiments. Table 2 shows an example of an email annotation where the annotators agreed to all but one segment.

Please click here for the full results of this evaluation. The following is a summary.

The initial κ for our model was calculated at 0.811 as opposed to 0.75 for the C&C model. This compares well with their earlier inter-annotator agreement experiment [10] that gave a value between 0.72 and 0.85, and is substantially lower than the one we achieved. To gather more insight into the major causes for disagreement, and make better comparisons with the C&C model, we decided to calculate further κ’s for both models. The κ where recomputed after the following considerations and the results are summarized in Table 3.

1. Given that we have the speech act (Deliver,Information,Ø) equivalent to their Deliver-Data and Deliver-Opinion speech acts, comparing this value with the one we obtained might be considered unfair. Thus we merged these categories for the C&C model. Although the resulting κ is now much closer to ours, we are satisfied with our result since even though we introduced a third parameter in our speech act definition to provide more structure to the knowledge representation, we virtually achieved the same inter-annotator agreement.
2. Disregarding the speech act object parameter for our model – in order to examine the effect of this parameter on κ.
3. Disregarding the speech act subject parameter for our model – for the same reason. In this case κ improved over the initial κ by a mere 0.004 of a fraction, suggesting that the introduction of the speech act subject parameter was successful.
4. All speech acts in the C&C model would have some effect on the semantic email client - except for Deliver-Data and Deliver-Opinion, which constitute 57.1% of the total category assignments for 57.2% of the total category assignments. After removing these categories, we obtained the relevant κ.
5. Similarly, (Deliver,Information,Ø), which accounts for 57.2% of the total category assignments was disregard for our model, obtained a relevant κ which is significantly higher than the C&C relevant κ.

Following these main results, we wanted to observe the causes for disagreements in order to improve our models. We were able to do this be creating and scrutinizing a confusion matrix for the disagreements between speech act assignments (see Figure 5). We then supervised the annotators with re-annotating the text segments which led to the disagreement only. This yielded an improved agreement of 0.609 (from total disagreement).

After considering all the feedback and results we decided on some changes to the sMail Speech Act Model. The most important was to introduce the Suggest action to cater for all weak-commisive, conditional and non-binding statements. Additionaly the Request action doubles for the Amend action, which was not physically included because its behavior is exactly like that for Request. The third change was adding the Abort action. The other two sMail models where appropriately updated to accomodate these changes.

Following the fine-tuning of the sMail Speech Act Model, we wanted to immediately evaluate any benefits that it might have on real data. The annotators where again instructed to reconsider the disagreements for the five highest-disagreeing categories with the new model in mind. The main result of this experiment is a higher kappa value of 0.732, significantly higher then the 0.609 achieved when we supervised the annotators with their annotation.

Conclusion: Our evaluation confirmed that despite adding further parameters to our Speech Act Model in order to be able to give more structure and semantics to email discourse, our model still resulted in a high-level of agreement between human annotators. Thus, we deem that our model is an improvement over previous taxonomies that modeled speech acts in the email domain.

[1] Carvalho, V., Cohen, W.: Improving Email Speech Act Analysis via N-gram Selection. HLT/NAACL, ACTS Workshop, New York City, NY (2006).

Phase II

The Speech Act Workflow Model has been evaluated indirectly through the evaluation of Semanta. Thus, this phase as planned here is redundant.

---

Future Work

---

We are currently dedicating the majority of our time to Semanta, being the proof of concept for this work. Testing and user experience with Semanta will possibly result in the further refinement of the sMail Conceptual Framework.

Besides striving to further refine our models, we are additionaly open to further extending our ideas and collaborating with other leading research in the area.

---

Publications

---

Simon Scerri, Siegfried Handschuh, Brian Davis
The path towards Semantic Email: Summary and Outlook.
In Proceedings of the Enhanced Messaging Workshop
Workshop at AAAI 2008 , Chicago, US, 2008.

Simon Scerri, Siegfried Handschuh, Stefan Decker
Semantic Email as a Communication Medium for the Social Semantic Desktop.
In Proceedings of the 5th European Semantic Web Conference
ESWC2008, Tenerife, Spain, 2008. VIDEO

Simon Scerri, Myriam Mencke, Brian Davis,
href="http://www.deri.ie/about/team/member/siegfried_handschuh/">Siegfried Handschuh
Evaluating the Ontology underlying sMail – the Conceptual Framework for Semantic Email Communication.
In Proceedings of the 6th International Conference on Language Resources and Evaluation
LREC2008, Marrakech, Morocco, 2008.

Simon Scerri, Brian Davis, Siegfried Handschuh
Improving Email Conversation Efficiency.
In Proceedings of the 6th International Workshop on Web Semantics (WebS 2007)
Workshop at DEXA2007, 2007.

Simon Scerri
Aiding the Workflow of Email Conversations by Enhancing Email with Semantics.
In PhD Symposium
Workshop at ESWC, Innsbruck, Austria, 2007.

---

Mailing List

---

Email the List:
semantic-email@lists.deri.org

Subscribe to the List::
Click Here to Subscribe

---

Contact

---

Simon Scerri (Main Contact)
Semantic Email Concept

Brian Davis
Human Language Technologies Adviser

Siegfried Handschuh
Scientific Supervisor