keras

Embedding

keras.layers.embeddings.Embedding(input_dim, output_dim, init='uniform', weights=None, W_regularizer=None, W_constraint=None)

Turn positive integers (indexes) into denses vectors of fixed size, eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]

• Input shape: 2D tensor with shape: (nb_samples, maxlen).

• Output shape: 3D tensor with shape: (nb_samples, maxlen, output_dim).

• Arguments:

  • input_dim: int >= 0. Size of the vocabulary, ie. 1+maximum integer index occuring in the input data.
  • output_dim: int >= 0. Dimension of the dense embedding.
  • init: name of initialization function for the weights of the layer (see: initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.
  • weights: list of numpy arrays to set as initial weights. The list should have 1 element, of shape (input_dim, output_dim).
  • W_regularizer: instance of the regularizers module (eg. L1 or L2 regularization), applied to the embedding matrix.
  • W_constraint: instance of the constraints module (eg. maxnorm, nonneg), applied to the embedding matrix.

WordContextProduct

keras.layers.embeddings.WordContextProduct(input_dim, proj_dim=128, 
        init='uniform', activation='sigmoid', weights=None)

This layer turns a pair of words (a pivot word + a context word, ie. a word from the same context as a pivot, or a random, out-of-context word), indentified by their indices in a vocabulary, into two dense reprensentations (word representation and context representation).

Then it returns activation(dot(pivot_embedding, context_embedding)), which can be trained to encode the probability of finding the context word in the context of the pivot word (or reciprocally depending on your training procedure).

For more context, see Mikolov et al.: Efficient Estimation of Word reprensentations in Vector Space

• Input shape: 2D tensor with shape: (nb_samples, 2).

• Output shape: 2D tensor with shape: (nb_samples, 1).

• Arguments:

  • input_dim: int >= 0. Size of the vocabulary, ie. 1+maximum integer index occuring in the input data.
  • proj_dim: int >= 0. Dimension of the dense embedding used internally.
  • init: name of initialization function for the embeddings (see: initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.
  • activation: name of activation function to use (see: activations), or alternatively, elementwise Theano function.
  • weights: list of numpy arrays to set as initial weights. The list should have 2 element, both of shape (input_dim, proj_dim). The first element is the word embedding weights, the second one is the context embedding weights.