Large MWorks datafiles

ereming's Avatar

ereming

21 Jul, 2014 03:48 PM

Hi Chris,

I just tested my program (the two reflection timing task), and I've discovered that the file sizes are very large. A run of 1000 trials is about 600mb without any eye data. I checked out the event stream and it looks like every attribute of every stimulus is being recorded each time the position of the moving stimulus changes. Is it possible to reduce the amount of non-changing stimulus information that's recorded?

Thanks,
Evan

  1. Support Staff 1 Posted by Christopher Sta... on 22 Jul, 2014 01:38 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    The only option you have for reducing the amount of stimulus announcement data is to disable individual stimulus announcements. To do this in recent nightlies, navigate to "MWServer -> Preferences -> Display" and uncheck "Announce stimuli individually via #announceStimulus events". Alternatively, in your setup_variables.xml, set the announce_individual_stimuli key of #mainScreenInfo to 0.

    Once this is done, stimuli will only be announced in #stimDisplayUpdate events.

    Cheers,
    Chris

  2. 2 Posted by ereming on 22 Jul, 2014 03:31 PM

    ereming's Avatar

    Hi Chris,

    That will be fine. Thanks!

    - Evan

  3. ereming closed this discussion on 22 Jul, 2014 03:31 PM.

  4. ereming re-opened this discussion on 23 Jul, 2014 08:15 PM

  5. 3 Posted by ereming on 23 Jul, 2014 08:42 PM

    ereming's Avatar

    Hi Chris,

    Turning off accounce_individual_stimuli shrinks the data file by about half, which still isn't really manageable. Testing the same program without the frame list stimuli further reduces the file size by about a factor of 15. Would it be feasible to program an option not to include information about display/stimulus updates driven by dynamic stimuli?

    - Evan

  6. Support Staff 4 Posted by Christopher Sta... on 25 Jul, 2014 06:18 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    Would it be feasible to program an option not to include information about display/stimulus updates driven by dynamic stimuli?

    Yes, we could do that. I think the right approach would be to distinguish between explicit updates (i.e. those initiated via an "Update Stimulus Display" action) and implicit updates (e.g. those requested by a dynamic stimulus). We could then provide a per-experiment setting to omit the list of stimulus parameters in #stimDisplayUpdate events for implicit updates. Hence, for implicit updates, display update events would include only a timestamp. Implementing this would be pretty easy.

    However, the availability of such an option is going to make some people (e.g. Jim) nervous. Having the event file record everything that happens in an experiment is an explicit design decision that dates back to the earliest days of MWorks. The reasoning is that the cost of running experiments is much, much greater than the cost of data storage, so it's better to store data you may not need than to run the risk of discarding data you do need (perhaps for some future, not-yet-thought-of analysis) and having to re-run your experiment.

    That said, I actually just chatted with Mehrdad about this, and he sounds comfortable with the approach I described. He made the argument that as long as you have the starting stimulus parameters and display update times, you can recreate the stimulus presentation after the fact. So if this approach sounds acceptable to you, I'll add it to my to-do list.

    Cheers,
    Chris

  7. 5 Posted by ereming on 25 Jul, 2014 07:31 PM

    ereming's Avatar

    I guess my response would be that the only thing I really care about is
    having manageable file sizes and at least being able to reconstruct the
    experiment as Mehrdad mentioned. How it's implemented isn't such a big
    deal, although I can think of a few different ways it could work. For
    instance, every #stimDisplayUpdate event contains a cell of *all* the the
    stimuli displayed on that update, along with *all *their parameters, even
    if only one parameter in one stimulus changed. In my case, there are about
    7-10 stimuli on the screen at a time, so if MWorks only logged the
    parameters of the changing stimulus, the size of the #stimulusDisplayUpdate
    data would be reduced by almost an order of magnitude, without losing any
    information.

    That being said, Mehrdad's solution works for me. But it does omit
    parameter information, which as you mentioned conflicts with one of the
    core MWorks philosophies.

    - Evan

  8. Support Staff 6 Posted by Christopher Sta... on 05 Sep, 2014 08:45 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    I think the right approach would be to distinguish between explicit updates (i.e. those initiated via an "Update Stimulus Display" action) and implicit updates (e.g. those requested by a dynamic stimulus). We could then provide a per-experiment setting to omit the list of stimulus parameters in #stimDisplayUpdate events for implicit updates.

    This solution is now implemented and will be available in the nightly build shortly.

    To disable stimulus parameter announcements for implicit display updates, your experiment must contain a stimulus_display device (which the editor includes by default in new experiments), and its announce_stimuli_on_implicit_updates parameter must be set to "NO". When emitting #stimDisplayUpdate events for implicit updates, MWorks will omit the list of stimulus parameters and instead report a single integer, which represents the number of stimuli displayed during that update.

    Hopefully, this will keep your event files down to a reasonable size. If you have questions or run in to any issues, please let me know.

    Cheers,
    Chris

  9. Christopher Stawarz closed this discussion on 17 Sep, 2014 02:06 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac