Editable nested fields in Writer
Stahl, Michael
Word fields are far more flexible than ODF/Writer fields - to most users, the additional expressiveness doesn't matter, but for some specialized use cases, Writer is limited. Writer had a rudimentary implementation of Word compatible fields for about 10 years, but it never worked well. We have added the ability to round-trip nested Word fields in RTF/DOCX formats, and fixed a bunch of long standing bugs along the way.
 Hello, my name is Michael Stahl. I work for a new company called Allotropia and I'm going to give you a status update about editable nested text fields in LibreOffice Writer. First, what are fields in Writer? A field is a magic character inside of paragraph text in the document model. At layout time, this magic character is then expanded to generate some piece of text. This generated text is basically plain text. It cannot have formatting inside or anything fancy like that. So it's a very simple mechanism basically to generate text. On the other hand in Word, fields are a bit more flexible. So a field is represented in two parts. It has a field instruction and a field result. Both of these parts are user-editable text that can have internal formatting applied to it. Other fancy things like paragraph breaks. You could put tables inside of either of these parts and you can even put fields inside of the field instruction or inside of the field result so you can have nested fields basically. The letter is very useful for conditional text for example. Word uses fields for all sorts of different things because they are so flexible. For example, an index like a table of content is a field in Word. Whereas in Writer, there are further more complex use cases. There are entirely separate objects in the document model. So there is some, an entire class hierarchy of indexes for example that have nothing to do with fields. So for most users, this difference is not all that important because they don't need the flexibility that Word fields provide. So it's just an implementation detail for most users. But some users need a bit more flexibility and want to edit, for example, the field instruction to get exactly the result that they want to achieve. So for those power users, Writer has been a bit limiting in that regard. So what is the situation in Writer with regard to the Word kind of fields? It turns out that many years ago already during OpenOffice.org times, an experimental implementation called fieldmarks was added, which is a representation of Word fields in Writer's document model. And it is implemented as a kind of bookmark. And the most important, there are several different kind of fieldmarks and the most important one is the text fieldmark. The way this works is that it has two dummy characters, one at the start and one at the end. And then between those dummy characters in the paragraph text, there is the field result. And the field instruction is not stored in the paragraph text. It is a separate string property that is associated with this fieldmark object. And obviously an obvious problem with this is that it cannot actually represent Word fields with full fidelity. Because as I said, Word fields can have any sort of content basically in the field instruction. And well, here it's just a plain text string. So, yeah, it was insufficient feature-wise and the implementation was also very renowned for having a lot of bugs and funny behaviors. So, yeah, that was basically the state we found when we started this work. So let's have a quick look at a class hierarchy diagram for these fieldmarks. And what you can see here is that the fieldmark implementation class is the one in the middle with the gray background. And it's derived from two base classes. The one is a mark base, which is the base class for all bookmark implementations. And the other one is an interface that is specific to fieldmarks. And then it has several subclasses. The one that interests us on which this entire talk is about is the text fieldmark. And then there are a couple other ones, like for example, the checkbox fieldmark, which is a bit different because it is not a range. In the paragraph text, it is just a single character and it can't do anything other than paint a checkbox. And then we have these two at the bottom, which are also a bit special because they essentially paint UI widgets in the document view. So for the drop-down, it would pop up a drop-down selection widget. And then there is this date fieldmark, which is an oddity. Let's put it that way because in Word, this is not actually a field. It's an SDT, a structured document tag. And those do not have a field instruction. So, yeah, so much for that. Now, what have we actually done about this? And all of this work is in the release 6.4. So we have first added a third dummy character for the field separator. So this means that we can now insert the field instruction also inside of the paragraph text. And with this, we can represent nested fields. We can have a fieldmark inside of the field instruction now. So that solves our major feature gap here, basically. And then we had to fix a lot of problems with these field characters being accidentally deleted while editing the text of the document, just by ordinary backspace or whatever. And, yeah, this is now fixed these sort of problems. Another issue was that for inserting or removing the fieldmark, there was no working undo and we fixed that too. And generally, there was a problem with the previous implementation with some bad programming practices, like there was lots of defensive programming to prevent the implementation from crashing when things were in an inconsistent state. But the problem with that is that it just makes it more difficult to fix bugs, basically. So we added lots of assertions to check when invariants are violated. And with the unit tests already and then again with the automated crash tests, we found lots of documents where crashes happened and we fixed them. And one particularly funny issue was that several import filters in Writer were able to insert control characters into the paragraph text. So Writer uses some magic characters for a special purpose, like its own text fields, for example. And, yeah, this could turn into an issue, so we are glad that we are now preventing this. So another thing we have done is we have added a configuration setting that basically forces all worm fields in RDF and.x import to be imported as field marks and not converted to writer fields. And this is not enabled by default because currently usability wise, it's usually better to convert the fields to writer fields because those can be expanded and it's more user friendly that way. But if you want to avoid any kind of data loss, then you can use this configuration setting. So that was for 6.4 and more recently we have added the ability to show or hide the field results and the field instruction, just like Word does. So you can use the field names menu item and this will be in the 7.1 release. And basically, you can see a screenshot on the left is Word with the field instruction on the top and the field result at the bottom. Yeah, well, the field result is actually some text I manually typed in for reasons that I'll explain later. And, yeah, on the right you can see the same document with writer and basically it looks the same now. And what remains to be done here is that currently writer is not able to actually expand the field instruction into a new field result. And this would be quite a bit of work. So, yeah, well, we hope that maybe someday we will have some funding to do this, but not currently. So, could also quickly look at this in writer itself. So this is this document and I can now select something and apply formatting to it or type something into it. And when I toggle this now, it shows the field result and this is also editable. And toggle back and we see the field instruction again. So, that was my presentation and thank you for listening.
 Hello, my name is Michael Stahl.  I work for a new company called Allotropia and I'm going to give you a status update  about editable nested text fields in LibreOffice Writer.  First, what are fields in Writer?  A field is a magic character inside of paragraph text in the document model.  At layout time, this magic character is then expanded to generate some piece of text.  This generated text is basically plain text. It cannot have formatting inside or anything fancy like that.  So it's a very simple mechanism basically to generate text.  On the other hand in Word, fields are a bit more flexible.  So a field is represented in two parts. It has a field instruction and a field result.  Both of these parts are user-editable text that can have internal formatting applied to it.  Other fancy things like paragraph breaks.  You could put tables inside of either of these parts and you can even put fields inside of the field instruction  or inside of the field result so you can have nested fields basically.  The letter is very useful for conditional text for example.  Word uses fields for all sorts of different things because they are so flexible.  For example, an index like a table of content is a field in Word.  Whereas in Writer, there are further more complex use cases.  There are entirely separate objects in the document model.  So there is some, an entire class hierarchy of indexes for example that have nothing to do with fields.  So for most users, this difference is not all that important because they don't need the flexibility that Word fields provide.  So it's just an implementation detail for most users.  But some users need a bit more flexibility and want to edit, for example, the field instruction to get exactly the result that they want to achieve.  So for those power users, Writer has been a bit limiting in that regard.  So what is the situation in Writer with regard to the Word kind of fields?  It turns out that many years ago already during OpenOffice.org times,  an experimental implementation called fieldmarks was added, which is a representation of Word fields in Writer's document model.  And it is implemented as a kind of bookmark.  And the most important, there are several different kind of fieldmarks and the most important one is the text fieldmark.  The way this works is that it has two dummy characters, one at the start and one at the end.  And then between those dummy characters in the paragraph text, there is the field result.  And the field instruction is not stored in the paragraph text.  It is a separate string property that is associated with this fieldmark object.  And obviously an obvious problem with this is that it cannot actually represent Word fields with full fidelity.  Because as I said, Word fields can have any sort of content basically in the field instruction.  And well, here it's just a plain text string.  So, yeah, it was insufficient feature-wise and the implementation was also very renowned for having a lot of bugs and funny behaviors.  So, yeah, that was basically the state we found when we started this work.  So let's have a quick look at a class hierarchy diagram for these fieldmarks.  And what you can see here is that the fieldmark implementation class is the one in the middle with the gray background.  And it's derived from two base classes.  The one is a mark base, which is the base class for all bookmark implementations.  And the other one is an interface that is specific to fieldmarks.  And then it has several subclasses.  The one that interests us on which this entire talk is about is the text fieldmark.  And then there are a couple other ones, like for example, the checkbox fieldmark, which is a bit different because it is not a range.  In the paragraph text, it is just a single character and it can't do anything other than paint a checkbox.  And then we have these two at the bottom, which are also a bit special because they essentially paint UI widgets in the document view.  So for the drop-down, it would pop up a drop-down selection widget.  And then there is this date fieldmark, which is an oddity.  Let's put it that way because in Word, this is not actually a field.  It's an SDT, a structured document tag.  And those do not have a field instruction.  So, yeah, so much for that.  Now, what have we actually done about this?  And all of this work is in the release 6.4.  So we have first added a third dummy character for the field separator.  So this means that we can now insert the field instruction also inside of the paragraph text.  And with this, we can represent nested fields.  We can have a fieldmark inside of the field instruction now.  So that solves our major feature gap here, basically.  And then we had to fix a lot of problems with these field characters being accidentally deleted while editing the text of the document,  just by ordinary backspace or whatever.  And, yeah, this is now fixed these sort of problems.  Another issue was that for inserting or removing the fieldmark, there was no working undo and we fixed that too.  And generally, there was a problem with the previous implementation with some bad programming practices,  like there was lots of defensive programming to prevent the implementation from crashing when things were in an inconsistent state.  But the problem with that is that it just makes it more difficult to fix bugs, basically.  So we added lots of assertions to check when invariants are violated.  And with the unit tests already and then again with the automated crash tests,  we found lots of documents where crashes happened and we fixed them.  And one particularly funny issue was that several import filters in Writer were able to insert control characters into the paragraph text.  So Writer uses some magic characters for a special purpose, like its own text fields, for example.  And, yeah, this could turn into an issue, so we are glad that we are now preventing this.  So another thing we have done is we have added a configuration setting that basically forces all worm fields in RDF and.x import  to be imported as field marks and not converted to writer fields.  And this is not enabled by default because currently usability wise, it's usually better to convert the fields to writer fields  because those can be expanded and it's more user friendly that way.  But if you want to avoid any kind of data loss, then you can use this configuration setting.  So that was for 6.4 and more recently we have added the ability to show or hide the field results and the field instruction,  just like Word does.  So you can use the field names menu item and this will be in the 7.1 release.  And basically, you can see a screenshot on the left is Word with the field instruction on the top and the field result at the bottom.  Yeah, well, the field result is actually some text I manually typed in for reasons that I'll explain later.  And, yeah, on the right you can see the same document with writer and basically it looks the same now.  And what remains to be done here is that currently writer is not able to actually expand the field instruction into a new field result.  And this would be quite a bit of work.  So, yeah, well, we hope that maybe someday we will have some funding to do this, but not currently.  So, could also quickly look at this in writer itself. So this is this document and I can now select something and apply formatting to it or type something into it.  And when I toggle this now, it shows the field result and this is also editable.  And toggle back and we see the field instruction again.  So, that was my presentation and thank you for listening.
lo_writernestedfields_0_179.png lo_writernestedfields_180_443.png lo_writernestedfields_444_491.png lo_writernestedfields_492_611.png lo_writernestedfields_612_5279.png lo_writernestedfields_5280_8771.png lo_writernestedfields_8772_12371.png lo_writernestedfields_12372_18887.png lo_writernestedfields_18888_19883.png lo_writernestedfields_19884_21107.png lo_writernestedfields_21108_21395.png lo_writernestedfields_21396_22019.png lo_writernestedfields_22020_22343.png lo_writernestedfields_22344_22391.png lo_writernestedfields_22392_22739.png lo_writernestedfields_22740_22799.png lo_writernestedfields_22800_22859.png lo_writernestedfields_22860_23243.png lo_writernestedfields_23244_23387.png lo_writernestedfields_23388_23531.png lo_writernestedfields_23532_23771.png lo_writernestedfields_23772_23855.png lo_writernestedfields_23856_24131.png lo_writernestedfields_24132_24258.png
Hello, my name is Michael Stahl. I work for a new company called Allotropia and I'm going to give you a status update about editable nested text fields in LibreOffice Writer. First, what are fields in Writer? A field is a magic character inside of paragraph text in the document model. At layout time, this magic character is then expanded to generate some piece of text. This generated text is basically plain text. It cannot have formatting inside or anything fancy like that. So it's a very simple mechanism basically to generate text. On the other hand in Word, fields are a bit more flexible. So a field is represented in two parts. It has a field instruction and a field result. Both of these parts are user-editable text that can have internal formatting applied to it. Other fancy things like paragraph breaks. You could put tables inside of either of these parts and you can even put fields inside of the field instruction or inside of the field result so you can have nested fields basically. The letter is very useful for conditional text for example. Word uses fields for all sorts of different things because they are so flexible. For example, an index like a table of content is a field in Word. Whereas in Writer, there are further more complex use cases. There are entirely separate objects in the document model. So there is some, an entire class hierarchy of indexes for example that have nothing to do with fields. So for most users, this difference is not all that important because they don't need the flexibility that Word fields provide. So it's just an implementation detail for most users. But some users need a bit more flexibility and want to edit, for example, the field instruction to get exactly the result that they want to achieve. So for those power users, Writer has been a bit limiting in that regard. So what is the situation in Writer with regard to the Word kind of fields? It turns out that many years ago already during OpenOffice.org times, an experimental implementation called fieldmarks was added, which is a representation of Word fields in Writer's document model. And it is implemented as a kind of bookmark. And the most important, there are several different kind of fieldmarks and the most important one is the text fieldmark. The way this works is that it has two dummy characters, one at the start and one at the end. And then between those dummy characters in the paragraph text, there is the field result. And the field instruction is not stored in the paragraph text. It is a separate string property that is associated with this fieldmark object. And obviously an obvious problem with this is that it cannot actually represent Word fields with full fidelity. Because as I said, Word fields can have any sort of content basically in the field instruction. And well, here it's just a plain text string. So, yeah, it was insufficient feature-wise and the implementation was also very renowned for having a lot of bugs and funny behaviors. So, yeah, that was basically the state we found when we started this work. So let's have a quick look at a class hierarchy diagram for these fieldmarks. And what you can see here is that the fieldmark implementation class is the one in the middle with the gray background. And it's derived from two base classes. The one is a mark base, which is the base class for all bookmark implementations. And the other one is an interface that is specific to fieldmarks. And then it has several subclasses. The one that interests us on which this entire talk is about is the text fieldmark. And then there are a couple other ones, like for example, the checkbox fieldmark, which is a bit different because it is not a range. In the paragraph text, it is just a single character and it can't do anything other than paint a checkbox. And then we have these two at the bottom, which are also a bit special because they essentially paint UI widgets in the document view. So for the drop-down, it would pop up a drop-down selection widget. And then there is this date fieldmark, which is an oddity. Let's put it that way because in Word, this is not actually a field. It's an SDT, a structured document tag. And those do not have a field instruction. So, yeah, so much for that. Now, what have we actually done about this? And all of this work is in the release 6.4. So we have first added a third dummy character for the field separator. So this means that we can now insert the field instruction also inside of the paragraph text. And with this, we can represent nested fields. We can have a fieldmark inside of the field instruction now. So that solves our major feature gap here, basically. And then we had to fix a lot of problems with these field characters being accidentally deleted while editing the text of the document, just by ordinary backspace or whatever. And, yeah, this is now fixed these sort of problems. Another issue was that for inserting or removing the fieldmark, there was no working undo and we fixed that too. And generally, there was a problem with the previous implementation with some bad programming practices, like there was lots of defensive programming to prevent the implementation from crashing when things were in an inconsistent state. But the problem with that is that it just makes it more difficult to fix bugs, basically. So we added lots of assertions to check when invariants are violated. And with the unit tests already and then again with the automated crash tests, we found lots of documents where crashes happened and we fixed them. And one particularly funny issue was that several import filters in Writer were able to insert control characters into the paragraph text. So Writer uses some magic characters for a special purpose, like its own text fields, for example. And, yeah, this could turn into an issue, so we are glad that we are now preventing this. So another thing we have done is we have added a configuration setting that basically forces all worm fields in RDF and.x import to be imported as field marks and not converted to writer fields. And this is not enabled by default because currently usability wise, it's usually better to convert the fields to writer fields because those can be expanded and it's more user friendly that way. But if you want to avoid any kind of data loss, then you can use this configuration setting. So that was for 6.4 and more recently we have added the ability to show or hide the field results and the field instruction, just like Word does. So you can use the field names menu item and this will be in the 7.1 release. And basically, you can see a screenshot on the left is Word with the field instruction on the top and the field result at the bottom. Yeah, well, the field result is actually some text I manually typed in for reasons that I'll explain later. And, yeah, on the right you can see the same document with writer and basically it looks the same now. And what remains to be done here is that currently writer is not able to actually expand the field instruction into a new field result. And this would be quite a bit of work. So, yeah, well, we hope that maybe someday we will have some funding to do this, but not currently. So, could also quickly look at this in writer itself. So this is this document and I can now select something and apply formatting to it or type something into it. And when I toggle this now, it shows the field result and this is also editable. And toggle back and we see the field instruction again. So, that was my presentation and thank you for listening.
fields are a bit more flexible. So a field is complex use cases. There are entirely separate detail for most users. But some users need a bit of Word fields in Writer's document model. And it nested fields. We can have a fieldmark inside of we have done is we have added a configuration the ability to show or hide the field results and