Changes between Version 2 and Version 3 of FdoTextReaderEnhancements

Timestamp:

Oct 18, 2007, 2:08:00 PM (17 years ago)

Author:

gregboone

Comment:

--

FdoTextReaderEnhancements

@@ -150 +150 @@
 }}}
 
-
-
+==== !FdoIoCachedStream ====
+
+A new stream class will be added:
+
+
+{{{
+class FdoIoCachedStream : public FdoIoStream
+{
+public:
+
+        FDO_API_COMMON static FdoIoCachedStream* Create( 
+                FdoIoStream* baseStream, 
+                FdoInt32 bufferSize=4096 
+        );
+
+        FDO_API_COMMON FdoIoStream* GetStream();
+
+        FDO_API_COMMON void Flush();
+};
+
+typedef FdoPtr<FdoIoCachedStream> FdoIoCachedStreamP;
+}}}
+
+
+''Parameters'':
+
+baseSteam: underlying stream where data will ultimately be read from or written to.
+
+bufferSize: size in bytes of data cache. The cache is used to provide better read/write performance
+
+''Description'':
+
+!GetStream() returns the underlying stream.
+
+Flush()  if the cache contains data to write, this function writes it to the underlying stream.
+
+This class provides buffered reads and writes from and to an FdoIStream by adding an in-memory cache.
+
+See '''Design Discussion''' below for the rationale behind this class.
+
+==== !FdoIoCachedFileStream  ====
+
+The following class will be added for convenience:
+
+
+{{{
+class FdoIoCachedFileStream : public FdoIoCachedStream
+{
+public:
+
+        FDO_API_COMMON static FdoIoCachedFileStream* Create(
+                FdoString* fileName, 
+                FdoString* accessModes, 
+                FdoInt32 bufferSize=4096 
+        );
+
+        FDO_API_COMMON static FdoIoCachedFileStream* Create(
+                FILE* fp,
+                FdoInt32 bufferSize=4096 
+        );
+};
+
+typedef FdoPtr<FdoIoCachedFileStream> FdoIoCachedFileStreamP;
+}}}
+
+The parameters are identical to those of !FdoIoFileStream and !FdoIoCachedStream. The behaviour is also identical except that reads and writes are buffered.
+
+!FdoIoCachedFileStream serves as a convenience class for wrapping an !FdoIoCachedStream around an !FdoIoFileStream to provide buffered access to a file. The following:
+
+{{{
+FdoIoCachedFileStreamP cfs = FdoIoCachedFileStream::Create(
+        L"myfile.txt",
+        L"r"
+);
+}}}
+
+is equivalent to:
+
+{{{
+FdoIoFileStreamP fs = FdoIoFileStream::Create(
+        L"myfile.txt",
+        L"r"
+);
+
+FdoIoCachedStreamP cfs = FdoIoCachedStream::Create( fs );
+}}}
+
+It might seem odd that the Create( FILE*, !FdoInt32) function would be needed since a FILE already provides buffered access. However, !FdoIoFileStream actually performs reads and writes through the underlying device for the FILE so the buffered access provided by FILE gets bypassed. An alternative approach would have been to leverage off the buffering provided by FILE. However, once !FdoIoCachedStream is in place, it can be used to implement buffered I/O for other types of devices.
+
+==== !FdoIoTextReader and !FdoIoTextWriter ====
+
+The following functions will have a slight behavioural change:
+
+{{{
+FdoIoTextReader::Create( FdoString* fileName )
+FdoIoTextWriter::Create( FdoString* fileName )
+}}}
+
+These functions used to automatically create an underlying stream of type !FdoIoFileStream. They will change to create the stream as an !FdoIoCachedFileStream. This will provide performance benefits.
+
+=== Design Discussion ===
+
+The Read functions, being added to !FdoIoTextReader, do not know in advance how much data they will read. Therefore, these functions must do one of two things:
+
+• read the data one byte at a time. [[br]]
+• read fixed sized chunks of data until the end of the string to read is reached.[[br]]
+
+For some devices (e.g. files) the second option provides better performance. However, it is much more complicated to implement since the end of the string to read can be in the middle of the current chunk. Therefore, the current read position must be reset from the end of the chunk to the middle. This cannot be done by changing the position on the underlying stream since not all streams support rewinding (some only support forward-only reading). Therefore the text reader would have to cache the remainder of the current chunk, to be read when the next string to read is requested. 
+
+The first option is much simpler but could be slow when reading from a file (the !FdoIoFileReader class does not provide any buffering on read). Therefore, the second option is preferred.
+
+Rather than complicate !FdoIoTextReader with the managing of a read buffer, this complication can be pushed down to a new class, called !FdoIoCachedStream. An added benefit is that doing the caching at the stream level allows buffered writing to file to be supported, providing performance benefits for streamed writing as well.
+
+When the first read request is made to !FdoIoCachedStream, it will read enough data from its base stream to fill its cache. For this read and subsequent reads, the data will be retrieved from the cache. When the end of the cache is reached, it will be flushed and filled again from the base stream. 
+
+When the first write request is made, the data is added to the cache. Subsequent writes will append to the cache. When the cache is full, it will be flushed by writing its contents to the base stream. 
+
+The actual algorithm for buffered reading and writing would be more complicated than described above, especially when mixed reads and writes are peformed. However, this document concentrates on the API, so the exact algorithm is beyond the scope of the document. To the caller, !FdoIoCacheStream will behave as if the reads and writes had been done directly against the base stream, except that they will be faster.
+
+Of the current streaming implementations, the file stream will benefit the most from caching. Caching would provide no benefit to the memory stream. For convenience, an !FdoIoCachedFileStream is provided. It allows a file stream, with cached stream wrapper, to be created in one step.
+
+=== More Examples ===
+
+The following reads each line of text from "myfile.txt"
+
+{{{
+FdoIoTextReaderP rdr = FdoIoTextReader( L"myfile.txt );
+FdoStringP line;
+
+while ( rdr->ReadLine(line) ) {
+        printf( "%ls\n", (FdoString*) line );
+}
+}}}
+
+The following does the same thing, but with a 50,000 byte read buffer:
+
+{{{
+FdoIoCachedFileStreamP stream = FdoIoCachedFileStream::Create( 
+        L"myfile.txt",
+        L"rt",
+        50000
+);
+
+FdoIoTextReaderP rdr = FdoIoTextReader( stream );
+FdoStringP line;
+
+while ( rdr->ReadLine(line) ) {
+        printf( "%ls\n", (FdoString*) line );
+}
+}}}
+
+=== Performance Stats ===
+
+The addition of the !FdoIoCachedStream class will help performance when reading or writing files. This will also translate into performance improvements for writing XML files. The reading of XML files will not be affected, since this is delegated to Xerces, which already does buffered reads. 
+
+The improvements on writing were verified by doing some tests against the !FdoIoTextWriter class. This class was used to write a 10Mb file on local disk, by multiple calls to FdoIoTextWriter::Write(). A number of tests were done, where each passed a different number of characters to each call to Write(). The following table shows the results:
+
+'''||# characters per call''''''||# calls to Write()'''||'''time (secs)'''||'''char/sec'''||
+||1||10000000||45||22222||
+||10||1000000||5.07||1972387||
+||20||500000||2.64||3787878||
+||100||100000||0.69||14492754||
+||4000||2500||0.28||35714286||
+||8000||1250||0.34||35714286||
+||50000||200||0.34||29411765||
+
+