FilterByTime keeps whole workspace Logs

Miscellaneous
1

Am again not sure if this is expected behaviour, or even an issue for others but thought it worth noting. Am also not sure if FilterByEvents is similarly affected

Applying FilterbyTime on a workspace appears to keep the logs from the entire length of the measurement, rather than filtering them solely to the time of interest. This can make for some large files. I had 1 day log runs, and was slicing it into ~200. Every slice was ~40Mb, but only contained 3*91 floats (X,Y,E). The file size reduced to 2Kb when logs were removed.

2

Hi Jos,
I’m wondering if this is specific to part of the sample logs. Are you able to send me the data file you mentioned in the post to daniel.murphy@stfc.ac.uk and I can do some investigating!

3

Try an INTER run 57582. This is one days worth of data. (I also sent you a script which acts on these, line 405 of this is used to remove the logs otherwise memory will fill up horribly, try deleting it and changing in the running script the time to a larger value)

4

Hi Jos,
As wonderful as your scripts are, they can be hard to figure out! As you said, I commented out line 405 that removed the logs and I ran your other script to produce one of your First_dayslice_##_of_864 Workspaces. I then used Show History to get a simple script of what you’re actually doing to make that Workspace.

FilterByTime is working correctly (Cuts Ch_ID_Err from 2847 to 4 entries), but your monitor workspace isn’t filtered so retains the sample logs for the whole run. then on line 8 you use AppendSpectra.

mon_rebin has the logs for the whole run whereas 57582_900_1000 has the filtered logs. When they are Appended, the data from the second is added to the first but the output workspace just has the sample logs of the first InputWorkspace. This is why day_slices end up with sample logs from the whole run.

Below is this simple ‘Show History’ script where I’ve added in CopyLogs on line 7. This is giving the mon_rebin the filtered logs and so they survive AppendSpectra. Try Running this script and see if this is the solution you’re looking for. If so then I hope you can find a way to work this into your INTER_autoreduction script! This line leaves First_dayslice_10_of_864 at 95 kB with logs (4 kB without logs).

Load(Filename='C:\\Path\to\TRANS.nxs', OutputWorkspace='TRANS')
Load(Filename=r'\\isis.cclrc.ac.uk\inst$\ndxinter\instrument\data\cycle_19_3\INTER00057582.nxs', OutputWorkspace='57582', LoadMonitors=True)
FilterByTime(InputWorkspace='57582', OutputWorkspace='57582_filter', StartTime=900, StopTime=1000)
Scale(InputWorkspace='57582_monitors', OutputWorkspace='mon_slice', Factor=0.0011684333374906953)
Rebin(InputWorkspace='57582_filter', OutputWorkspace='57582_900_1000', Params='0,20,100000', PreserveEvents=False)
Rebin(InputWorkspace='mon_slice', OutputWorkspace='mon_rebin', Params='0,20,100000', PreserveEvents=False)
CopyLogs('57582_900_1000','mon_rebin','WipeExisting') # Replace mon_rebin logs with the filtered logs
AppendSpectra(InputWorkspace1='mon_rebin', InputWorkspace2='57582_900_1000', OutputWorkspace='57582_900_1000')
ReflectometryReductionOneAuto(InputWorkspace='57582_900_1000', ThetaIn=0.64981704415018859, WavelengthMin=1, WavelengthMax=17, FirstTransmissionRun='TRANS', MomentumTransferMin=0.0083852242849074519, MomentumTransferStep=0.014999999999999999, MomentumTransferMax=0.1425151372438889, ScaleFactor=1, OutputWorkspaceBinned='First_dayslice_10_of_864', OutputWorkspace='First_dayslice_10_of_864_unbinned', OutputWorkspaceWavelength='57582_900_1000_lam')