An objective function (or loss function, or optimization score function) is one of the two parameters required to compile a model:
model.compile(loss='mean_squared_error', optimizer='sgd')
You can either pass the name of an existing objective, or pass a Theano symbolic function that returns a scalar and takes the following two arguments:
For a few examples of such functions, check out the objectives source.
(nb_samples, nb_classes)
.