BizTalk Pipeline Out Of Memory
I’ve developed many custom components for BizTalk in the past several years, but I have never had to work with very large input files (>100MB), primarily I’ve simply created output files. In a recent need to gather several years worth of historical data, it became apparent that memory constraints were going to be pushed to their peak. After battling the frequent “System.OutOfMemoryException” it was time to hunt for the solution.
The resolution to the BizTalk memory problem relied in the source structure (MemoryStream) we were using for storing our stream for the IBaseMessage.
MemoryStream revisedStream = new MemoryStream();
The discussion helpfully points out that we should be using the VirtualStream with a few caveats. The trick to get the VirtualStream object is to include the source from the BizTalk server samples (<BizTalk Install Path>\SDK\Samples\Pipelines\SchemaResolverComponent\SchemaResolverFlatFileDasm) or use the Microsoft.BizTalk.Streaming.dll as it exists in the GAC (<Windows Path>\assembly\GAC_MSIL\Microsoft.BizTalk.Streaming\3.0.1.0__31bf3856ad364e35\). I chose the later approach, which required adding a reference to Microsoft.BizTalk.Streaming.dll to my custom pipeline solution. I first copied the assembly from the GAC to the <biztalk install path> for future solutions where I may need to reuse the same assembly source. After adding the reference via Visual Studio, it was simply a matter of changing the MemoryStream reference to VirtualStream.
VirtualStream revisedStream = new VirtualStream();
Be sure you rewind the stream before assigning the bodyparts data.
// Rewind the stream so it is ready for use by the messaging engine revisedStream.Seek(0, SeekOrigin.Begin);
So why does this solution work? Behind the scenes, VirtualStream is wrapping excessive data (user defined, but defaults to >4MB) to temporary disk using a BufferedStream. This is why we can avoid the “System.OutOfMemoryException” – since the VirtualStream will offload the large data stream to disk I/O instead of purely storing our IBaseMessage in RAM (via MemoryStream). Yes, there’s certainly overhead depending on where your temporary directory resides, but in most cases this will be the solution you’re looking for. MSDN docs has other BizTalk specific streams if you want to dive deeper.
Tags: BizTalk, Pipeline Components, VirtualStream
