LotusScript Replace a memory hog?
Category show-n-tell thursday
I rarely crash servers. Correction: my code rarely crashes servers... I once wrote an agent that scaled very poorly and eventually required 3 hours to successfully complete once the data it had to process had grown unmanageable over the course of about 2 years of use. I subsequently rewrote it to use a more efficient approach, and now it takes about two minutes to run on the same volume of data. But until the other day I don't think I've ever actually crashed a server with my code. Ironically, what finally did it was the use of a technique I hadn't considered very resource-intensive: passing LotusScript's Replace() function a 25 index array. Like legions of other Dominoheads, I was relieved when Replace was added to our arsenal, and now use it prolifically. But generally I use it to search a single string for occurrences of a second string and replace it with a third... despite the knowledge that the documented parameters are arrays. Well, as it turns out, it doesn't seem to like arrays much.
I have a database containing about 200 documents that I want to convert to a different field structure. While I'm converting the data anyway, I might as well clean it up a bit. So I identified 25 patterns that I simply wanted to remove wherever found. So I figured I'd be all clever and stuff, and do the removal as follows:
Let varUnwantedData = Split("pattern1,pattern2,pattern3,pattern4........,pattern25",",")
Let strNewFieldValue = Replace(docCurrent.GetItemValue("OldFieldName")(0),varUnwantedData,"")
Call docCurrent.ReplaceItemValue("NewFieldName",strCleanData)
Call docCurrent.RemoveItem("OldFieldName")
Seems alright, don't it? Grab the old field value, replace all instances of any unwanted data with an empty string, and set the new field's value to the returned String. Yeah, well, not so much. See, what Replace is doing behind the scenes to determine the String value is looping through each index of the array, and doing a Replace of each index's value. Sort of a hidden Forall. Which is fine; that's precisely what I want it to do. The trouble seems to lie in the lack of a named container for the result to be stored each time. So it keeps all of 'em. Shouldn't have to... it should be able to forget what it originally was each time it loops back through. But it doesn't. So it's eventually (if it makes it all the way through) holding 26 copies of the data in memory: the original value, plus all variations resulting from the individual replacement operations. Assuming this completes successfully, then it realizes that it can store the last one in the specified variable and discard the rest. But it's too late... the server's already tanked.
I changed the code to the following:
Let varUnwantedData = Split("pattern1,pattern2,pattern3,pattern4........,pattern25",",")
Let strNewFieldValue = docCurrent.GetItemValue("OldFieldName")(0)
Forall strUnwantedData in varUnwantedData
Let strNewFieldValue = Replace(strNewFieldValue,strUnwantedData ,"")
End Forall
Call docCurrent.ReplaceItemValue("NewFieldName",strNewFieldValue)
Call docCurrent.RemoveItem("OldFieldName")
Now it completes the replacement on all four of the fields that I apply this to on all 200 documents - in addition to all of the conversion it was already doing before I decided I wanted to add this cleanup step - in about 30 seconds.
I rarely crash servers. Correction: my code rarely crashes servers... I once wrote an agent that scaled very poorly and eventually required 3 hours to successfully complete once the data it had to process had grown unmanageable over the course of about 2 years of use. I subsequently rewrote it to use a more efficient approach, and now it takes about two minutes to run on the same volume of data. But until the other day I don't think I've ever actually crashed a server with my code. Ironically, what finally did it was the use of a technique I hadn't considered very resource-intensive: passing LotusScript's Replace() function a 25 index array. Like legions of other Dominoheads, I was relieved when Replace was added to our arsenal, and now use it prolifically. But generally I use it to search a single string for occurrences of a second string and replace it with a third... despite the knowledge that the documented parameters are arrays. Well, as it turns out, it doesn't seem to like arrays much.
I have a database containing about 200 documents that I want to convert to a different field structure. While I'm converting the data anyway, I might as well clean it up a bit. So I identified 25 patterns that I simply wanted to remove wherever found. So I figured I'd be all clever and stuff, and do the removal as follows:
Let varUnwantedData = Split("pattern1,pattern2,pattern3,pattern4........,pattern25",",")
Let strNewFieldValue = Replace(docCurrent.GetItemValue("OldFieldName")(0),varUnwantedData,"")
Call docCurrent.ReplaceItemValue("NewFieldName",strCleanData)
Call docCurrent.RemoveItem("OldFieldName")
Seems alright, don't it? Grab the old field value, replace all instances of any unwanted data with an empty string, and set the new field's value to the returned String. Yeah, well, not so much. See, what Replace is doing behind the scenes to determine the String value is looping through each index of the array, and doing a Replace of each index's value. Sort of a hidden Forall. Which is fine; that's precisely what I want it to do. The trouble seems to lie in the lack of a named container for the result to be stored each time. So it keeps all of 'em. Shouldn't have to... it should be able to forget what it originally was each time it loops back through. But it doesn't. So it's eventually (if it makes it all the way through) holding 26 copies of the data in memory: the original value, plus all variations resulting from the individual replacement operations. Assuming this completes successfully, then it realizes that it can store the last one in the specified variable and discard the rest. But it's too late... the server's already tanked.
I changed the code to the following:
Let varUnwantedData = Split("pattern1,pattern2,pattern3,pattern4........,pattern25",",")
Let strNewFieldValue = docCurrent.GetItemValue("OldFieldName")(0)
Forall strUnwantedData in varUnwantedData
Let strNewFieldValue = Replace(strNewFieldValue,strUnwantedData ,"")
End Forall
Call docCurrent.ReplaceItemValue("NewFieldName",strNewFieldValue)
Call docCurrent.RemoveItem("OldFieldName")
Now it completes the replacement on all four of the fields that I apply this to on all 200 documents - in addition to all of the conversion it was already doing before I decided I wanted to add this cleanup step - in about 30 seconds.
