importing MWK to Matlab using Python

meyert11's Avatar

meyert11

25 Jul, 2021 02:17 AM

Hey Chris,

so I'm trying to replace our importing function that brings mwk variables in and saves them to a mat file using Python. Much of the process was actually pretty easy (we can load/save mat files in Python with scipy, much of the matlab code is very similar to Python format), however I'm hung up on saving the data because the f_getevents brings the data in as a funky data structure. When I bring the import the mwk data into a variable with f.get_events(code...) it will create an 'eventwrapper' that looks like a dictionary within a dictionary, called '<mworks._mworks.EventWrapper at 0x7faa26d2bed0>' If I put that into X, I can go to X[0].data, X[1].data ... and get information, but if I try to save X to a mat file (scipi.io.savemat) it is empty.

If you have any sample Python scripts for importing mwk data, perhaps that would be the easiest way to see what works for handling these even wrappers.

  1. 1 Posted by meyert11 on 25 Jul, 2021 06:48 PM

    meyert11's Avatar

    <update>
    I've continued looking into this and I'm wondering if the confusion is based on the new mwk2 file formatting (as I read in the Current Event File Format (MWK2)) section of the manual. I was expecting a nested dictionary format similar to numpy, however this is in SQLite (which I'm less familiar with). I'm wondering if it's this new formatting that is making the porting of the output of the f_getevents then importing to a dictionary for the .mat file to such funky formatting.
    I'll look up how to handle SQlite data, and hopefully there's a simple way to move it to numpy (then .mat via scipi).

  2. Support Staff 2 Posted by Christopher Sta... on 26 Jul, 2021 03:04 PM

    Christopher Stawarz's Avatar

    Hi Travis,

    MWKFile's get_events method returns a sequence of event objects. The event objects (and, prior to MWorks 0.10, the sequence itself) are a custom type, so scipi.io.savemat isn't going to know what to do with them. In any case, the docs for savemat say that you need to provide a dictionary mapping names to variable values, so passing in any sort of non-mapping type isn't going to work.

    The closest I can get to successfully using savemat is this function:

    def events_to_mwk(infile, outfile):
        with MWKFile(infile) as fp:
            evts = [{'event_code': e.code, 'time_us': e.time, 'data': e.data}
                    for e in fp.get_events_iter()
                    if e.data is not None]
        scipy.io.savemat(outfile, {'events': evts}, long_field_names=True)
    

    However, it isn't fully successful, because savemat seems unable to save Python dictionaries that can't be converted to MATLAB struct's. In particular, MWorks codec events can't be saved, as they map integers to dictionaries, and integers aren't valid struct keys in MATLAB. (MWorks' own MATLAB data tools will convert dictionary values that can't be MATLAB structs into Map instances.)

    So, I don't think your desired approach is going to work. But if you're looking to store MWorks events in MAT files, why wouldn't you just use MATLAB to do so? And if you want your events in a format that can be read in both Python and MATLAB, why not just stick with MWK2? If you can explain what you're trying to accomplish, I may be able to suggest a more fruitful approach.

    Cheers,
    Chris

  3. 3 Posted by meyert11 on 26 Jul, 2021 04:50 PM

    meyert11's Avatar

    Hi Chris,

    Yeah sorry I didn't give much context before. We'd like to start to move
    away from Matlab to Python, Mworks looked like a good place to start since
    it supports both; So I started rewriting our importing function (which
    currently uses Matlab) to Python, but this is going to be a transition, so
    I'd love to make a temporary bridge so we can still use all of our Matlab
    functions (redundancy always seems to be the best failsafe :)

    Your method might be good enough for the data we extract so I'll give that
    a try. I was trying to avoid bringing the mworks code into Python, then
    rebuilding the dictionaries from scratch before porting it over with scipi
    - but that might be the inevitable next step. One trick that might come up
    is there are some System dictionaries that are nested, like
    #stimDisplayUpdate.

    To answer your question: why not just keep it in mwk2 - From what I've seen
    so far it's kind of cumbersome to extract data from these files because it
    seems to be in serialization. I'm used to extracting data straight from
    databases into arrays, not serialized data formats so it's a less intuitive
    process. I'm not a computer science major so perhaps there's a gap in my
    knowledge that would make this more intuitive that I just haven't come
    across :) In fact the MWK2 being in SQlite now might be my savior, but I've
    got a queue of SQlite python youtube tutorials now to watch haha.

    Travis

  4. 4 Posted by meyert11 on 26 Jul, 2021 07:16 PM

    meyert11's Avatar

    Hi Chris,

    Your code was very helpful, I think I'm getting close! I'm going to include
    my python code I'm working on to help give an idea of the issue, but I
    think it's a resolvable issue!

    In the code I create a list of codecs I want to extract from the mwk2 file
    ('called list). I then create a codecs array that includes all codecs in
    the file with codecs = f.codec. I scan through the codecs list and if a
    member equals a member in my list, I try to extract it into a variable. if
    I try extracting the data like the manual says 'data =
    f.get_events(codes=[codecs[i]])' then the data is in a weird wrapper,
    however if I do it like in your code it saves it as a list of dictionaries;
    ' data = [{'event_code': e.code, 'time_us': e.time, 'data': e.data} for e
    in f.get_events_iter() if e.code == i]'

    it's a bit clunky because the iter will run through each event so I'm sure
    there's a more efficient way to code this up, but I'd be fine with a
    working code. What I'm trying to do now is to automate the variable 'data'
    to equal the event code by evaluating a string, where the variable changes,
    name = [codecs[i]]
    st_out = str(name[0] + """ = [{'event_code': e.code, 'time_us': e.time,
    'data': e.data} for e in f.get_events_iter() if e.code == i]""")
    then just eval(st_out). It's not working because of a syntax error
    (pointing to the equal sign) but I'll work on figuring this out. I just
    figured providing this info would give you a better idea of how I'm trying
    to extract the variable info from the mwk2 and port it. My thinking is I'll
    create a list for each event code, then just save all of them with scipy
    and then it should be in the same format for matlab.

    T

  5. 5 Posted by meyert11 on 26 Jul, 2021 09:19 PM

    meyert11's Avatar

    Hi Chris,

    <update> Success! Sort of...
    Ok, figured some stuff out. Using your code I can extract the Events in the
    appropriate way! The: sort of... scipy doesn't allow appending more than
    one dictionary to a present dictionary, Yikes! So at this point I think I
    have the solution, but I'll just have to come up with a change that we're
    going to use, maybe I'll just store all of our Event lists as one nested
    dictionary that In Matlab we can just extract to test whether it will work
    with our current code. I thought I'd upload the current code, which runs
    (with an mwk2 file called velvet) but the saved mat file is
    registered corrupt in Matlab, which many people online have had the exact
    some problem with this appending method. So it's sort of a fix, but unless
    you see an obvious solution, you can close the case. Thanks for your help!

    T

  6. Support Staff 6 Posted by Christopher Sta... on 27 Jul, 2021 08:57 PM

    Christopher Stawarz's Avatar

    Hi Travis,

    Try this:

    data = {}
    
    with MWKFile('velvet.mwk2') as f:
        rc = f.reverse_codec
        for name in getEventList():
            code = rc[name]
            if name.startswith('#'):
                name = name[1:]
            data[name] = [{'event_code': e.code, 'time_us': e.time, 'data': e.data}
                          for e in f.get_events_iter(codes=[code])]
    
    io.savemat('velvet_mwk.mat', data)
    

    I haven't run that code, so there may be typos, but it should do what you want and get everything in one file.

    Cheers,
    Chris

  7. 7 Posted by meyert11 on 28 Jul, 2021 11:58 AM

    meyert11's Avatar

    Hi Chris,

    Your intuition is spot on! I just tried that yesterday and it worked, so
    scipy doesn't save the dictionary as one nested dictionary, it unwraps it
    saving it as all of the internal dictionaries, which is perfect!

    It's a bit slow so it'd be nice not to have to iterate with f.get_events_iter
     I wonder why the 'f.get_events(codec) spits out something different;
    because it'd be more efficient to use that I'd think.

    Matlab can bring it in and it's all <cells> whereas using f.get_events
    saves some things as <struct> so that's the last thing I'm working on and
    then we'll have a Python Matlab Bridge woo hoo!!

    T

  8. 8 Posted by meyert11 on 06 Aug, 2021 03:48 PM

    meyert11's Avatar

    Hi Chris,

    I'm making a ton of headway on this, I'm able to bring in the mwk2 file using the MWK2Raeader class, and moving, saving the data in a numpy array, then saving into a matlab file! Better yet, it's OS independent, which clears out an old post I had on here years ago, woo hoo!

    one quick question - using the MWK2Reader, is there a way to get the dictionary of all the codes and their corresponding variables? In the old way, a call to the codec gave this list, 'f.codec' and I assume the sqlite table should have this information somewhere in the mwk2 file.

  9. Support Staff 9 Posted by Christopher Sta... on 09 Aug, 2021 01:37 PM

    Christopher Stawarz's Avatar

    Hi Travis,

    using the MWK2Reader, is there a way to get the dictionary of all the codes and their corresponding variables?

    Yes. See this post.

    Better yet, it's OS independent, which clears out an old post I had on here years ago, woo hoo!

    Yeah, MWK2Reader is proving to be popular, mostly for that reason. I'm planning to make something like it part of MWorks' official data tools. Not sure when I'll get to it, but hopefully it'll be relatively soon.

    Cheers,
    Chris

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac