🤟🏼Natural Language Processing Unit 6 – Vector Semantics and Embeddings

What's the Big Idea?

• Vector semantics and embeddings represent words, phrases, or documents as dense vectors in a high-dimensional space
• Captures semantic relationships between words based on their co-occurrence patterns in large text corpora
• Enables machines to understand and reason about the meaning of words and their relationships
• Fundamental concept in natural language processing (NLP) tasks such as text classification, sentiment analysis, and machine translation
• Allows for mathematical operations on word vectors to uncover semantic relationships (king - man + woman ≈ queen)
• Provides a continuous representation of words, allowing for capturing nuanced relationships and similarities
• Enables transfer learning by pre-training embeddings on large unlabeled text corpora and fine-tuning on specific tasks

Key Concepts to Know

• Word embeddings: Dense vector representations of words that capture their semantic and syntactic properties
  • Each word is represented as a fixed-length vector (typically 100-300 dimensions)
  • Words with similar meanings have similar vector representations
• Distributional hypothesis: Words that occur in similar contexts tend to have similar meanings
• Co-occurrence matrix: Captures the frequency of words appearing together in a specific context window
  • Rows represent target words, and columns represent context words
  • Used as input to generate word embeddings
• Dimensionality reduction techniques (Singular Value Decomposition, Principal Component Analysis) compress co-occurrence matrix into dense word vectors
• Cosine similarity measures the similarity between two word vectors based on the angle between them
• Word2Vec: Popular algorithm for learning word embeddings using shallow neural networks
  • Consists of two architectures: Continuous Bag-of-Words (CBOW) and Skip-Gram
• GloVe (Global Vectors for Word Representation): Another widely used algorithm for learning word embeddings based on global word-word co-occurrence statistics

The Math Behind It

• Word embeddings are learned by optimizing an objective function that captures the relationship between words and their contexts
• Skip-Gram model maximizes the probability of predicting the context words given a target word:

$\text{maximize} \sum_{t=1}^{T} \sum_{-c \leq j \leq c, j \neq 0} \log p(w_{t+j} | w_t)$

• Continuous Bag-of-Words (CBOW) model predicts the target word given its context:

$\text{maximize} \sum_{t=1}^{T} \log p(w_t | w_{t-c}, \ldots, w_{t-1}, w_{t+1}, \ldots, w_{t+c})$

• Softmax function is used to calculate the probability of a word given its context:

$p(w_O | w_I) = \frac{\exp(v_{w_O}^T v_{w_I})}{\sum_{w=1}^{W} \exp(v_w^T v_{w_I})}$

• Negative sampling is used to efficiently approximate the softmax function by sampling negative examples
• GloVe model learns word vectors by minimizing the difference between the dot product of word vectors and the logarithm of their co-occurrence probability:

$J = \sum_{i,j=1}^{V} f(X_{ij}) (w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij})^2$

How It Works in Practice

• Preprocess text data by tokenizing, lowercasing, and removing stop words and punctuation
• Construct a co-occurrence matrix by sliding a context window over the text and counting word co-occurrences
• Apply dimensionality reduction techniques (SVD, PCA) to the co-occurrence matrix to obtain dense word vectors
• Alternatively, use neural network-based models (Word2Vec, GloVe) to learn word embeddings directly from text data
  • Train the models on large text corpora (Wikipedia, news articles, books) to capture general word relationships
  • Fine-tune the pre-trained embeddings on task-specific data for better performance
• Evaluate the quality of word embeddings using intrinsic and extrinsic evaluation methods
  • Intrinsic evaluation: Word similarity tasks, analogy tasks (man : king :: woman : queen)
  • Extrinsic evaluation: Downstream NLP tasks (text classification, named entity recognition)
• Visualize word embeddings using dimensionality reduction techniques (t-SNE, PCA) to understand the learned semantic relationships

Tools and Technologies

• Popular libraries for working with word embeddings:
  • Gensim: Python library for topic modeling and document similarity retrieval
    • Provides implementations of Word2Vec, FastText, and other embedding models
  • SpaCy: Industrial-strength natural language processing library in Python
    • Offers pre-trained word embeddings and easy integration with downstream NLP tasks
  • TensorFlow and Keras: Deep learning frameworks with built-in support for training and using word embeddings
• Pre-trained word embeddings:
  • Word2Vec embeddings trained on Google News corpus (300 dimensions)
  • GloVe embeddings trained on Wikipedia and Gigaword corpus (50, 100, 200, 300 dimensions)
  • FastText embeddings trained on Wikipedia and Common Crawl (300 dimensions)
• Visualization tools:
  • TensorBoard: Visualization toolkit for TensorFlow models, including embedding projector
  • Embedding Projector: Web-based tool for visualizing high-dimensional data, including word embeddings

Real-World Applications

• Text classification: Represent documents as averages or weighted averages of word embeddings for sentiment analysis, topic classification, or spam detection
• Named entity recognition (NER): Use word embeddings as input features to identify and classify named entities (persons, organizations, locations) in text
• Machine translation: Incorporate word embeddings to capture semantic relationships between words in source and target languages
• Text summarization: Leverage word embeddings to identify important sentences or phrases based on their semantic similarity to the document's main topics
• Recommendation systems: Use word embeddings to find similar items or products based on their textual descriptions
• Question answering: Employ word embeddings to measure the semantic similarity between questions and potential answers
• Text generation: Utilize word embeddings as input to language models for generating coherent and semantically meaningful text

Challenges and Limitations

• Word embeddings capture biases present in the training data, which can perpetuate stereotypes or discriminatory associations
  • Addressing bias requires careful data curation and debiasing techniques
• Polysemy and homonymy: Word embeddings struggle to handle words with multiple meanings (polysemy) or different words with the same spelling (homonymy)
  • Contextual word embeddings (ELMo, BERT) aim to address this by generating context-specific representations
• Out-of-vocabulary (OOV) words: Word embeddings cannot provide meaningful representations for words not seen during training
  • Subword embeddings (FastText) or character-level embeddings can mitigate this issue
• Lack of interpretability: Word embeddings are dense, continuous vectors that are difficult to interpret directly
  • Techniques like nearest neighbor analysis or dimensionality reduction can provide some insights into the learned relationships
• Domain-specific terminology: Word embeddings trained on general corpora may not capture the nuances of domain-specific language
  • Training embeddings on domain-specific text or fine-tuning pre-trained embeddings can improve performance

What's Next?

• Contextual word embeddings (ELMo, BERT) generate dynamic representations based on the surrounding context, addressing polysemy and homonymy issues
• Transformer-based models (BERT, GPT) leverage attention mechanisms to capture long-range dependencies and generate powerful language representations
• Cross-lingual word embeddings enable knowledge transfer between languages and support multilingual NLP tasks
• Multimodal embeddings combine text, images, and other modalities to learn richer representations
• Graph embeddings extend the concept of embeddings to graph-structured data, enabling tasks like node classification and link prediction
• Adapting word embeddings to handle evolving language and emerging terminology remains an ongoing challenge
• Developing more interpretable and explainable word embedding models is an active area of research
• Incorporating knowledge graphs and semantic networks to enhance word embeddings with structured knowledge