Virtual Humans Thoughts

This page covers some random thoughts on different aspects of virtual humans that interest me, and which I can hopefully find time to follow up on in papers and experimentation. They roughly align with the boxes in our diagram in the Virtual Humans book.

 

 

More information on many of these is in my recent whitepaper on Chatbots - Past, Present and Future.

Interfacing

I've always thought that, as the diagram above suggests, the VH "brain" should be completely separate to the VH "body". This means that the brain can have access to as much computing power as it needs, and can be replicated and sharded for survivability, and can run multiple bodies (which include robots and 2D chat windows and 3D avatars) at the same time. In order to enable this I've always assumed (and implemented in SL) that there are two interface standards running between the body and the brain:

• Avatar Sensory Markup Language (ASML) - for the sense "organs" in the body to let the brain know what it is sensing; and
• Avatar Action Markup Language (AAML) - for the brain to tell the "body" what to do, including what to say, how to emote, how to gesture, how to move etc.

Memory

I've always liked the idea of modelling VH memory in a similar way to how human memories appears to be segmented. First this means split between "working" memory - the 5-7 things pertinent to the current task(s) and "long term" memory (everything else). Long term memory is then split into:

• Semantic Memory - our knowledge of tangible and intangible "things", stored as triples in a knowledge graph so connections can be readily explored and used to drive human-style conversations. A Top Level Ontology provides the high level structure, essentially a game of 20 questions from "thing" and "tangible"/"intangible" through animal/vegetable/mineral to all the things we think about on a day to day basis;
• Episodic Memory - all the things that have happened to us - again stored as triples in a knowledge graph, and tightly bound with the semantic memory (they are parts of the same graph), and each episode holding relevant emotions and valency which can then be used to control how ell it is remembered, or how easily forgotten or hidden; and
• Procedural Memory - how to do things - a bit cumbersome to store as triples but probably has the meta-data stored as triples as part of the knowledge graph and then the procedures defined in something like XML or event Typescript.

 

Emotions

I’m continuing to use and develop the E-AI model that I used with Sarah Slater at Wolverhampton University.

The basic model is:

• Emotion Alert Database (EAD) - what emotions any thing generates
• Any stimulus has high-road check against EAD for response (action, emotion, physio, exclamation)
• Same stimulus has low-road pass to the main NL/response processing system
• EAD response reduces with over stimulation (accommodation)
• Emotion response decays if no further trigger
• Things that please or hurt the VH add to/enhance EAD reaction for future events

Ideally the EAD is stored as triples within a knowledge graph, rather than in a separate database.

Emotions are based around Eckman’s 5 (Happy, Sad, Anger, Disgust, Fear) plus surprise, plus a range of Secondary emotions in a PAD (pleasure-arousal-dominance) space, with half-life type decays. Instant emotions drive mood, also mapped in a PAD space.

Motivation

For simplicity I’ll probably continue to use Maslow’s hierarchy of needs as a model for the motivation of the VH.

• Self-actualisation
• Esteem
• Social & Belonging
• Safety
• Physiological

The VH will regularly monitor the gaps between the desired and actual levels of each need, and take action accordingly. However there needs to be a random and non-optimal approach to task switching/need fulfilment so that the VH can, to an extent, get so engrossed in high level need A but ignore a growing urgency to fulfil more low level/foundational need B (e.g. accidentally pulling an all-nighter).

Internal Narrative

When thinking about "consciousness" and "sentience" one of the most important elements to me is whether or not the entity has an internal narrative. We as humans (or at least me!) seem to have a continuous internal dialogue going on, think about ourselves, what we are doing, our emotions, other people, the future, the past etc. For me a true virtual human also needs that internal narrative. That is something which is quite hard to implement in a Q&A based text chatbot, even an LLM. In contrast it is very easy in a virtual world, as the VH continually exists, continually has things happening around them, and continually has the opportunity for action and interaction. Our Halo bot "lived" in SL. She was on 24/7, but followed SL's 4 hour day cycle, so was awake for 3 hours doing things around our island and then slept for an hour. Whilst we never got as far as a full internal dialogue she was continually responding to things, or following her own goals (based on the motivation model above) and then even blogged about her day every evening on the LiveJournal site!

 

Halo on our island in SL - just by the Twitter fountain

Dreaming

I think that the idea of Dreams within Virtual Humans is fascinating, and closely related to Memory and Internal Narrative. There are two/three particular aspects I’m interested in:

• How dreams are generated when the VH “sorts” its memories from the day whilst asleep
• How dreams can be used to rehearse awake period actions (which allies with day-dreams)
• How dreams could actually take place in a virtual world, so the VH literally lives them out in a (different) virtual world and, like us, cannot tell that they are not real whilst they are happening.