Using Linguistic Annotations in Statistical MAchine Translation of Film Subtitles, WSJO, translation [ Pobierz całość w formacie PDF ]
Using Linguistic Annotations in
Statistical Machine Translation of Film Subtitles
Christian Hardmeier
Fondazione Bruno Kessler
Human Language Technologies
Via Sommarive, 18
I-38050 Povo (Trento)
hardmeier@fbk.eu
Martin Volk
Universitat Z urich
Inst. f ur Computerlinguistik
Binzm uhlestrasse 14
CH-8050 Z urich
volk@cl.uzh.ch
Abstract
Statistical Machine Translation (SMT) has
been successfully employed to support
translation of film subtitles. We explore
the integration of Constraint Grammar
corpus annotations into a Swedish–Danish
subtitle SMT system in the framework of
factored SMT. While the usefulness of the
annotations is limited with large amounts
of parallel data, we show that linguistic an-
notations can increase the gains in transla-
tion quality when monolingual data in the
target language is added to an SMT system
based on a small parallel corpus.
1 Introduction
1 million subtitles of the training corpus used by
Volk and Harder was morphologically annotated
with the DanGram parser (Bick, 2001). We in-
tegrated the annotations into the translation pro-
cess using the methods of factored Statistical Ma-
chine Translation (Koehn and Hoang, 2007) im-
plemented in the widely used Moses software. Af-
ter describing the corpus data and giving a short
overview over the methods used, we present a
number of experiments comparing different fac-
tored SMT setups. The experiments are then repli-
cated with reduced training corpora which contain
only part of the available training data. These se-
ries of experiments provide insights about the im-
pact of corpus size on the effectivity of using lin-
guistic abstractions for SMT.
In countries where foreign-language films and se-
ries on television are routinely subtitled rather than
dubbed, there is a considerable demand for effi-
ciently produced subtitle translations. Although
superficially it may seem that subtitles are not ap-
propriate for automatic processing as a result of
their literary character, it turns out that their typi-
cal text structure, characterised by brevity and syn-
tactic simplicity, and the immense text volumes
processed daily by specialised subtitling compa-
nies make it possible to produce raw translations
of film subtitles with statistical methods quite ef-
fectively. If these raw translations are subse-
quently post-edited by skilled staff, production
quality translations can be obtained with consider-
ably less effort than if the subtitles were translated
by human translators with no computer assistance.
A successful subtitle Machine Translation sys-
tem for the language pair Swedish–Danish, which
has now entered into productive use, has been pre-
sented by Volk and Harder (2007). The goal of the
present study is to explore whether and how the
quality of a Statistical Machine Translation (SMT)
system of film subtitles can be improved by us-
ing linguistic annotations. To this end, a subset of
2 Machine translation of subtitles
As a text genre, subtitles play a curious role in
a complex environment of different media and
modalities. They depend on the medium film,
which combines a visual channel with an audi-
tive component composed of spoken language and
non-linguistic elements such as noise or music.
Within this framework, they render the spoken di-
alogue into written text, are blended in with the vi-
sual channel and displayed simultaneously as the
original sound track is played back, which redun-
dantly contains the same information in a form
that may or may not be accessible to the viewer.
In their linguistic form, subtitles should be faith-
ful, both in contents and in style, to the film dia-
logue which they represent. This means in partic-
ular that they usually try to convey an impression
of orality. On the other hand, they are constrained
by the mode of their presentation: short, written
captions superimposed on the picture frame.
According to Becquemont (1996), the charac-
teristics of subtitles are governed by the inter-
play of two conflicting principles:
unobtrusive-
ness
(discretion) and
readability
(lisibilite). In
KristiinaJokinenandEckhardBick(Eds.)
NODALIDA2009ConferenceProceedings,pp.57–64
 ChristianHardmeierandMartinVolk
order to provide a satisfactory experience to the
viewers, it is paramount that the subtitles help
them quickly understand the meaning of the dia-
logue without distracting them from enjoying the
film. The amount of text that can be displayed at
one time is limited by the area of the screen that
may be covered by subtitles (usually no more than
two lines) and by the minimum time the subtitle
must remain on screen to ensure that it can actually
be read. As a result, the subtitle text must be short-
ened with respect to the full dialogue text in the
actors’ script. The extent of the reduction depends
on the script and on the exact limitations imposed
for a specific subtitling task, but may amount to
as much as 30 % and reach 50 % in extreme cases
(Tomaszkiewicz, 1993, 6).
trusiveness and readability, they are not very fre-
quent.
It is worth noting that, unlike rule-based Ma-
chine Translation systems, a statistical system
does not in general have any difficulties translat-
ing ungrammatical or fragmentary input: phrase-
based SMT, operating entirely on the level of
words and word sequences, does not require the
input to be amenable to any particular kind of lin-
guistic analysis such as parsing. Whilst this ap-
proach makes it difficult to handle some linguistic
challenges such as long-distance dependencies, it
has the advantage of making the system more ro-
bust to unexpected input, which is more important
for subtitles.
We have only been able to sketch the character-
istics of the subtitle text genre in this paper. Dıaz-
Cintas and Remael (2007) provide a detailed intro-
duction, including the linguistics of subtitling and
translation issues, and Pedersen (2007) discusses
the peculiarities of subtitling in Scandinavia.
As a result of this processing and the consid-
erations underlying it, subtitles have a number of
properties that make them especially well suited
for Statistical Machine Translation. Owing to their
presentational constraints, they mainly consist of
comparatively short and simple phrases. Current
SMT systems, when trained on a sufficient amount
of data, have reliable ways of handling word trans-
lation and local structure. By contrast, they are
still fairly weak at modelling long-range depen-
dencies and reordering. Compared to other text
genres, this weakness is less of an issue in the Sta-
tistical Machine Translation of subtitles thanks to
their brevity and simple structure. Indeed, half
of the subtitles in the Swedish part of our par-
allel training corpus are no more than 11 tokens
long, including two tokens to mark the beginning
and the end of the segment and counting every
punctuation mark as a separate token. A consider-
able number of subtitles only contains one or two
words, besides punctuation, often consisting en-
tirely of a few words of affirmation, negation or
abuse. These subtitles can easily be translated by
an SMT system that has seen similar examples be-
fore.
3 Constraint Grammar annotations
To explore the potential of linguistically annotated
data, our complete subtitle corpus, both in Danish
and in Swedish, was linguistically analysed with
the DanGram Constraint Grammar (CG) parser
(Bick, 2001), a system originally developed for
the analysis of Danish for which there is also a
Swedish grammar. Constraint Grammar (Karls-
son, 1990) is a formalism for natural language
parsing. Conceptually, a CG parser first produces
possible analyses for each word by considering its
morphological features and then applies constrain-
ing rules to filter out analyses that do not fit into
the context. Thus, the word forms are gradually
disambiguated, until only one analysis remains;
multiple analyses may be retained if the sentence
is ambiguous.
The annotations produced by the DanGram
parser were output as tags attached to individual
words as in the following example:
The orientation of the genre towards spoken lan-
guage also has some disadvantages for Machine
Translation systems. It is possible that the lan-
guage of the subtitles, influenced by characteris-
tics of speech, contains unexpected features such
as stutterings, word repetitions or renderings of
non-standard pronunciations that confuse the sys-
tem. Such features are occasionally employed by
subtitlers to lend additional colour to the text, but
as they are in stark conflict with the ideals of unob-
$-
Vad[vad]<interr>INDPNEUSNOM@ACC>
vet[veta]<mv>VPRAKT@FS-QUE
du [du]PERS2SUTRSNOM@<SUBJ
om [om]PRP@<PIV
det[den]<dem>PERSNEU3SACC@P<
$?
In addition to the word forms and the accompany-
ing lemmas (in square brackets), the annotations
58
 UsingLinguisticAnnotationsinStatisticalMachineTranslationofFilmSubtitles
contained part-of-speech (POS) tags such as
INDP
for “independent pronoun” or
V
for “verb”, a mor-
phological analysis for each word (such as
NEUS
NOM
for “neuter singular nominative”) and a tag
specifying the syntactic function of the word in
the sentence (such as
@ACC>
, indicating that the
sentence-initial pronoun is an accusative object of
the following verb). For some words, more fine-
grained part-of-speech information was specified
in angle brackets, such as
<interr>
for “interrog-
ative pronoun” or
<mv>
for “verb of movement”.
In our experiments, we used word forms, lemmas,
POS tags and morphological analyses. The fine-
grained POS tags and the syntax tags were not
used.
language requires a different word order, reorder-
ing is possible at the cost of a score penalty. The
translation model has no notion of sequence, so
it cannot control reordering. The language model
can, but it has no access to the source language
text, so it considers word order only from the point
of view of TL grammaticality and cannot model
systematic differences in word order between two
languages.
Lexical reordering models
(Koehn et
al., 2005) address this issue in a more explicit way
by modelling the probability of certain changes in
word order, such as swapping words, conditioned
on the source and target language phrase pair that
is being processed.
In its basic form, Statistical Machine Transla-
tion treats word tokens as atomic and does not
permit further decomposition or access to single
features of the words. Factored SMT (Koehn and
Hoang, 2007) extends this model by represent-
ing words as vectors composed of a number of
features and makes it possible to integrate word-
level annotations such as those produced by a Con-
straint Grammar parser into the translation pro-
cess. The individual components of the feature
vectors are called
factors
. In order to map be-
tween different factors on the target language side,
the Moses decoder works with
generation mod-
els
, which are implemented as dictionaries and ex-
tracted from the target-language side of the train-
ing corpus. They can be used, e. g., to generate
word forms from lemmas and morphology tags, or
to transform word forms into part-of-speech tags,
which could then be checked using a language
model.
4 Factored Statistical Machine
Translation
Statistical Machine Translation formalises the
translation process by modelling the probabilities
of target language (TL) output strings
T
given a
source language (SL) input string
S
,
p
(
T|S
), and
conducting a search for the output string
T
with
the highest probability. In the Moses decoder
(Koehn et al., 2007), which we used in our exper-
iments, this probability is decomposed into a log-
linear combination of a number of feature func-
tions
h
i
(
S,T
), which map a pair of a source and a
target language element to a score based on differ-
ent submodels such as translation models or lan-
guage models. Each feature function is associated
with a weight l
i
that specifies its contribution to
the overall score:
T
=arg max
T
log
p
(
T|S
)
5 Experiments with the full corpus
=arg max
T
å
i
l
i
h
i
(
S,T
)
We ran three series of experiments to study the
effects of different SMT system setups on trans-
lation quality with three different configurations
of training corpus sizes. For each condition, sev-
eral Statistical Machine Translation systems were
trained and evaluated.
In the
full data
condition, the complete system
was trained on a parallel corpus of some 900,000
subtitles with source language Swedish and target
language Danish, corresponding to around 10 mil-
lion tokens in each language. The feature weights
were optimised using minimum error rate train-
ing (Och, 2003) on a development set of 1,000
subtitles that had not been used for training, then
the system was evaluated on a 10,000 subtitle test
The
translation models
employed in factored
SMT are phrase-based. The phrases included in
a translation model are extracted from a word-
aligned parallel corpus with the techniques de-
scribed by Koehn et al. (2003). The associated
probabilities are estimated by the relative frequen-
cies of the extracted phrase pairs in the same cor-
pus. For
language modelling
, we used the SRILM
toolkit (Stolcke, 2002); unless otherwise specified,
6-gram language models with modified Kneser-
Ney smoothing were used.
The SMT decoder tries to translate the words
and phrases of the source language sentence in the
order in which they occur in the input. If the target
59
 ChristianHardmeierandMartinVolk
set that had been held out during the whole de-
velopment phase. The translations were evalu-
ated with the widely used BLEU and NIST scores
(Papineni et al., 2002; Doddington, 2002). The
outcomes of different experiments were compared
with a randomisation-based hypothesis test (Co-
hen, 1995, 165–177). The test was two-sided, and
the confidence level was fixed at 95 %.
The results of the experiments can be found in
table 1. The baseline system used only a transla-
tion model operating on word forms and a 6-gram
language model on word forms. This is a stan-
dard setup for an unfactored SMT system. Two
systems additionally included a 6-gram language
model operating on part-of-speech tags and a 5-
gram language model operating on morphology
tags, respectively. The annotation factors required
by these language models were produced from the
word forms by suitable generation models.
In the
full data
condition, both the part-
of-speech and the morphology language model
brought a slight, but statistically significant gain
in terms of BLEU scores, which indicates that
abstract information about grammar can in some
cases help the SMT system choose the right words.
The improvement is small; indeed, it is not re-
flected in the NIST scores, but some beneficial ef-
fects of the additional language models can be ob-
served in the individual output sentences.
One thing that can be achieved by taking word
class information into account is the disambigua-
tion of ambiguous word forms. Consider the fol-
lowing example:
ditional language models helped to rule out this
error and correctly translate
mitt emot
as
over for
,
yielding a much better translation. Neither of them
output the adverb
lige
‘just’ found in the reference
translation, for which there is no explicit equiva-
lent in the input sentence.
In the next example, the POS and the morphol-
ogy language model produced different output:
Input:
Daliga kontrakt, dalig ledning, daliga agenter.
Reference:
Darlige kontrakter, darlig styring, darlige
agenter.
Baseline:
Darlige kontrakt, darlig forbindelse, darlige
agenter.
POS:
Darlige kontrakt, darlig ledelse, darlige agenter.
Morphology:
Darlige kontrakter, darlig forbindelse,
darlige agenter.
In Swedish, the indefinite singular and plu-
ral forms of the word
kontrakt
‘contract(s)’ are
homonymous. The two SMT systems without sup-
port for morphological analysis incorrectly pro-
duced the singular form of the noun in Danish.
The morphology language model recognised that
the plural adjective
darlige
‘bad’ is more likely
to be followed by a plural noun and preferred
the correct Danish plural form
kontrakter
‘con-
tracts’. The different translations of the word
ledning
as ‘management’ or ‘connection’ can be
pinned down to a subtle influence of the generation
model probability estimates. They illustrate how
sensitive the system output is in the face of true
ambiguity. None of the systems presented here has
the capability of reliably choosing the right word
based on the context in this case.
In three experiments, the baseline configuration
was extended by adding lexical reordering mod-
els conditioned on word forms, lemmas and part-
of-speech tags, respectively. As in the language
model experiments, the required annotation fac-
tors on the TL side were produced by generation
models.
The lexical reordering models turn out to be
useful in the
full data
experiments only when con-
ditioned on word forms. When conditioned on
lemmas, the score is not significantly different
from the baseline score, and when conditioned on
part-of-speech tags, it is significantly lower. In this
case, the most valuable information for lexical re-
ordering lies in the word form itself. Lemma and
part of speech are obviously not the right abstrac-
tions to model the reordering processes when suf-
ficient data is available.
Input:
Ingen vill bo mitt emot en ismaskin.
Reference:
Ingen vil bo lige over for en ismaskine.
Baseline:
Ingen vil bo mit imod en ismaskin.
POS/Morphology:
Ingen vil bo over for en ismaskin.
Since the word
ismaskin
‘ice machine’ does not
occur in the Swedish part of the training corpus,
none of the SMT systems was able to translate it.
All of them copied the Swedish input word liter-
ally to the output, which is a mistake that cannot
be fixed by a language model. However, there is a
clear difference in the translation of the phrase
mitt
emot
‘opposite’. For some reason, the baseline
system chose to translate the two words separately
and mistakenly interpreted the adverb
mitt
, which
is part of the Swedish expression, as the homony-
mous first person neuter possessive pronoun ‘my’,
translating the Swedish phrase as ungrammatical
Danish
mit imod
‘my against’. Both of the ad-
60
 UsingLinguisticAnnotationsinStatisticalMachineTranslationofFilmSubtitles
Table 1 Experimental results
full data
symmetric
asymmetric
BLEU NIST
BLEU NIST
BLEU NIST
Baseline
53.67 % 8.18
42.12 % 6.83
44.85 % 7.10
Language models
parts of speech
?
53.90 % 8.17
?
42.59 % 6.87 44.71 % 7.08
morphology
?
54.07 % 8.18
?
42.86 % 6.92
?
44.95 % 7.09
Lexical reordering
word forms
?
53.99 % 8.21
42.13 % 6.83
44.72 % 7.05
lemmas
53.59 % 8.15
?
42.30 % 6.86
44.71 % 7.06
parts of speech
53.36 % 8.13
?
42.33 % 6.86
44.63 % 7.05
Analytical translation
53.73 % 8.18
?
42.28 % 6.90
?
46.73 % 7.34
?
BLEU score significantly above baseline (
p<.
05)
BLEU score significantly below baseline (
p<.
05)
Another system, which we call the
analytical
translation
system, was modelled on suggestions
by Koehn and Hoang (2007) and Bojar (2007). It
used the lemmas and the output of the morpholog-
ical analysis to decompose the translation process
and use separate components to handle the transfer
of lexical and grammatical information. In order
to achieve this, the baseline system was extended
with additional translation tables mapping SL lem-
mas to TL lemmas and SL morphology tags to TL
morphology tags, respectively. In the target lan-
guage, a generation model was used to transform
lemmas and morphology tags into word forms.
The results reported by Koehn and Hoang (2007)
strongly indicate that this translation approach is
not sufficient on its own; instead, the decomposed
translation approach should be combined with a
standard word form translation model so that one
can be used in those cases where the other fails.
This configuration was therefore adopted for our
experiments.
The analytical translation approach fails to
achieve any significant score improvement with
the full parallel corpus. Closer examination of
the MT output reveals that the strategy of using
lemmas and morphological information to trans-
late unknown word forms works in principle, as
shown by the following example:
Input:
Molly har visat mig br ollopsfotona.
Reference:
Molly har vist mig fotoene fra brylluppet.
Baseline:
Molly har vist mig br ollopsfotona.
Analytical:
Molly har vist mig bryllupsbillederne.
In this sentence, there can be no doubt that the out-
put produced by the analytical system is superior
to that of the baseline system. Where the base-
line system copied the Swedish word
brollops-
fotona
‘wedding photos’ literally into the Dan-
ish text, the translation found by the analytical
model,
bryllupsbillederne
‘wedding pictures’, is
both semantically and syntactically flawless. Un-
fortunately, the reference translation uses different
words, so the evaluation scores will not reflect this
improvement.
The lack of success of analytical translation in
terms of evaluation scores can be ascribed to at
least three factors: Firstly, there are relatively few
vocabulary gaps in our data, which is due to the
size of training corpus. Only 1.19 % (1,311 of
109,823) of the input tokens are tagged as un-
known by the decoder in the baseline system. As
a result, there is not much room for improvement
with an approach specifically designed to handle
vocabulary coverage, especially if this approach
itself fails in some of the cases missed by the base-
line system: Analytical translation brings this fig-
ure down to 0.88 % (970 tokens), but no further.
Secondly, employing generation tables trained on
the same corpus as the translation tables used by
the system limits the attainable gains from the out-
set, since a required word form that is not found in
the translation table is likely to be missing from
the generation table, too. Thirdly, in case of vo-
cabulary gaps in the translation tables, chances
are that the system will not be able to produce
the optimal translation for the input sentence. In-
stead, an approach like analytical translation aims
61
[ Pobierz całość w formacie PDF ]