Time Decay of Information
The below is a snapshot in time of my evolving thought process on how to deal with information aging out in learning models. I may periodically refresh that thinking in place or extend to a new post.
(Textual) Information
Information aging out for learning (human and machine) requires thinking about the problem space from at least two different angles: 1.) Embeddings 2.) Weights / Models
Of which there are three scenarios: 1.) Explicitly dated information 2.) Implicity dated information 3.) Undated information
Of sources across a different set of vectors: a.) Mostly Trusted b.) Untrusted
from multiple sources including, but not limited to:
- Books (our oldest form of information) - Permanent form of information
- Articles (news, blogs, journals) - Semipermanent form of information
- Social Media (the most ephemeral form of information)
In addition, we need to consider whether the model or human is aiming for general or deep knowledge of the topic at hand.
Deep knowledge may have less stickiness over time, while general knowledge may be more resilient to time decay.
Finally, we need to look at how that knowledge could decay over time. Could an initially 2048-dim embedding decay into a 128-dim embedding? This "semantic evaporation" could be a mechanism for long-term memory management, where detailed nuances are pruned while the core concept (the "centroid") remains.
Multimodal Information?
A lot of the public work that I've read has gone into single-mode learning study, whereas humans do not learn (well) from text-only. Even if one is "book smart," the structures that we use to retain and process information really on mapping to other concepts.
Textual Models like GLiNER (Generalist Named Entity Recognition) offer hints as to how that linkage might be established. Lead author Urchade Zaratiana and the team are pushing this into GLiNER2, which aims for unified, schema-driven information extraction across multiple tasks like NER, classification, and structured data extraction.
Future Work: Proving out the ideas
To prove out these ideas, we're looking at "Temporal Embeddings"—vector spaces that incorporate a time-decay function directly into the similarity calculation. This prioritizes recent information without entirely discarding historical context, much like how human memory functions. We also need to explore GLiNER's viability in zero-shot aspect-based sentiment analysis as a way to track changing sentiments over time.