Next steps with MWorks and Python

mhisted's Avatar

mhisted

19 Jun, 2017 06:36 PM

Hi Chris,

I'm doing some refactoring of our python experiment code now, so I'm thinking a bit more about MWorks and Python.

It's now been a number of years since I sent the below, and you've implemented many of these. 1 and 2 below have turned out to be very helpful for us.

I have one small and one larger point to ask about.

Small: did you ever get to my third bullet below -- a Python conduit function that returns the current state of a variable? Right now I register a callback for every variable and record all of their values, in Python. That would probably be better done in C++ on the Python side of the conduit (as I assume the client does with the stream of events from the server). Do you agree?

Large: Before I dig into Python over the next few days, I want to consider my lab's next steps with MWorks and Python over the next 1-3 years. I see three options.
1. Keep slogging along along with the Python bridge/conduit.
2. Put our Python code inside MWServer (using the feature you implemented to let MWServer run Python code).
3. Think about ways to run experiments, and define and call MWorks objects/actions from Python.

Details, pros/cons:
1. Keep slogging along along with the Python bridge/conduit.
Pro: Minimal engineering work for me up front
Con: Latency between Client and Python is long (50 ms with a sometimes longer tail). Lots of variable callbacks, putting load on the single-threaded Py interpreter. (fixable, see small pt above). Syncing XML and Python is cumbersome and errorprone.
Still have to deal with non-Python syntax and parser, which slows down our development.

2. Put our Python code inside MWServer (using the feature you implemented to let MWServer run Python code).
Pro: Hopefully Python call latency is much shorter (but needs testing). Variable callback processing load goes away.
Con: Engineering up front for me. Need to figure out system python packages, install and sync across machines. Might Python need to be called at parse time? If so need to work with Chris. Have to debug timing - will all Python dispatch and simple statements be as fast as the XML?
Still have to deal with non-Python syntax in XML and parser.

3. Think about ways to run expts and call MWorks objects from Python
Con: Engineering up front for me. Chris and Jim don't really want to do this. Even if they agree, takes potentially a lot of Chris's time (my lab can deal with a few months delay, though). Python syntax not a great fit for state machine. Have to debug timing - will all Python dispatch and simple statements be as fast as the XML?
Pro: Can probably use anaconda, which is newer and easier to sync across machines. Can use Python syntax to do conditionals, variable assignment, loops, subfunctions, numpy -- makes our experiments much more maintainable and readable. Latency issues go away (but timing issues same as (2)).

Any comments? Happy to talk. Arash if you'd like to be part of the convo please let us know.

Mark

  1. Support Staff 1 Posted by Christopher Sta... on 20 Jun, 2017 08:30 PM

    Christopher Stawarz's Avatar

    Hi Mark,

    did you ever get to my third bullet below -- a Python conduit function that returns the current state of a variable? Right now I register a callback for every variable and record all of their values, in Python. That would probably be better done in C++ on the Python side of the conduit (as I assume the client does with the stream of events from the server). Do you agree?

    No, I haven't addressed this yet. I still agree that it would be useful.

    That said, I've begun to think we should expose the core Client class in Python. That would give you everything you have now with the conduits, plus variable-value caching, plus full control to load/start/stop/unload/etc. the experiment. For client-side Python scripts, it would also eliminate the intermediate hop through MWClient, since the script could communicate directly with MWServer.

    Before I dig into Python over the next few days, I want to consider my lab's next steps with MWorks and Python over the next 1-3 years. I see three options.

    As a first step, I think it's worth reviewing what you're currently using Python for and whether some or all of it can now be handled directly by MWorks.

    I know you don't want to use MWEditor or edit XML by hand. I fully empathize and agree with that position, which is why I've been working hard on an alternative experiment format/DSL. (I know I've mentioned this before, and I swear I'm going to share details about it very soon!)

    But beyond that, what do you currently do in Python that you can't or don't want to do directly in your experiment XML? As I recall, the lack of array and dictionary support was an issue for you, but I've addressed many of MWorks' previous limitations in that area. Are you using Python to communicate with I/O devices? Something else?

    Chris

  2. 2 Posted by mhisted on 22 Jun, 2017 04:43 PM

    mhisted's Avatar

    Hi Chris,

    You're asking about why a DSL wouldn't be sufficient, and why I'm asking
    for the experiment to be specified in python rather than a DSL/XML. First,
    let me say I propose Python would replace the XML fully transparently from
    a client/server UI perspective. So there would be a button to "Load
    experiment", mw.report() in python would display on the server console, and
    events generated by mw.setVariable() would get passed to event streams just
    as currently. No plotting would be allowed from the experiment.

    > But beyond that, what do you currently do in Python that you can't or don't
    > want to do directly in your experiment XML? As I recall, the lack of array
    > and dictionary support was an issue for you, but I've addressed many of
    > MWorks' previous limitations in that area. Are you using Python to
    > communicate with I/O devices? Something else?

    I looked back over some of our past emails about this, and the issues
    python would now improve over a DSL are:
    - subroutines (i.e. code to send a set of digital events, that needs to be
    called from multiple states, where flexibility is required that can't be
    done in C++)
    - sharing code across experiments: Python modules do this already. You and
    I would need to work together to implement and test includes in the DSL.
    That requires some hard thinking about scoping. Already solved and
    documented in Python.
    - local variables. Our experiments have variables like "debugMs" "debugUs"
    "tempUs" all over the place to put values into reports. Python already has
    a way to declare such variables in local scope. MWorks may do it too, but
    it would take some time for me to ask you about that and possibly for you
    to change it.
    - report actions. Limited in MWorks, but the Python string processor is
    sophisticated, and well documented and could be used to flexibly specify
    strings for reporting.
    - operators. You and I spent time discussing truncating integer division
    in MWorks and the change in identity of the '/' operator in MW 0.6. That
    operation for all types is fully specified in Python, and that would have
    saved us time over discussing and reimplementing in MWorks.

    - More generally, object types and syntax are fully specified in Python. I
    see a lot of pain for you and also myself fixing all the corner cases of a
    DSL syntax, when this work has already been done for extant languages. You
    and I have spent time on, from the best of my memory:

    - if/else
    - variable literals and how their parsing depends on the "type" attribute
    - operators, including / and // (I would also prefer to use bitshift
    operators but it's kind of not worth my time now to work with you on that
    since I have workaround)
    - list/dict access (in the future we might want .append(); already
    specified and tested in Python)
    - variable types and casting
    - trial selection / selection variables which I finally moved to Python
    because it's just more flexible and fits our needs

    Some of the above are solved, but I think switching to Python would prevent
    dealing with these kind of corner cases in the future and ultimately save
    us time. For example, in the future I could see wanting for loops (esp.
    useful for arrays/dicts) or transitions that depend on both timers and a
    condition (currently worked around with '&& timestamp >= now()' in our
    transitions. I guess I just see a DSL or future XML as a long treadmill of
    continuing work by you and me to work out syntax that is now solved by
    Python.

    What do you see as the major downsides of specifying experiments in
    Python? I know you mentioned state machine syntax. Agree it would be
    unwieldy, but the upsides outweigh this. And I noticed recently the
    package "transitions" on github has solved some of these problems.

    best,
    Mark

  3. 3 Posted by mhisted on 24 Jun, 2017 01:31 AM

    mhisted's Avatar

    Hi Chris and Arash,

    Arash and I spoke recently about the future of MWorks (and I recently
    chatted with John Maunsell as well).
    I'm planning to get in touch with Jim about charting our labs' plans with
    MWorks.
    Chris, are you ok with me forwarding this conversation along?

    thanks,
    Mark

  4. Support Staff 4 Posted by Christopher Sta... on 24 Jun, 2017 01:53 PM

    Christopher Stawarz's Avatar

    Hi Mark,

    I'm happy to open this discussion up to anyone who's interested. I've made the thread public, so you can just share the URL.

    However, I would appreciate it if, before making any decisions or jumping to conclusions, you'd give me a chance to present my position more clearly and to respond directly to your arguments. I'm working on some examples to demonstrate why I think the DSL I've developed provides a better user experience than Python-based experiments. I will try to finish those and get back to you early next week.

    In the meantime, I'll just make a few quick points:

    1. The DSL I keep mentioning isn't just an idea. It's done -- as in, I'm currently writing full experiments in it and running them via MWClient and MWServer. The implementation is still in a private branch, but only because I have a few loose ends to tie up before I perform integration testing and move it in to the nightly build.

      Of course, by "done", I don't mean that I'm not open to feedback, or that I'm unwilling to make changes to it. I hope for lots of feedback, and I'm sure things will change. My point is simply that we're not talking about a hypothetical DSL. It's a real thing, with a real, working implementation.

      Also, from here on out I'll refer to it by its name, MWEL, which is an acronym for "MWorks Experiment Language".

    2. When run, MWEL experiments are first converted into MWorks XML and are then parsed and executed like any other experiment. This has significant advantages over a Python-based approach. Specifically, (1) it doesn't introduce a new, independent codepath for implementing experiments that needs to be developed and maintained, and (2) it ensures that XML-based experiments (by which I mean every MWorks experiment ever written until now) remain fully supported, first-class citizens in the MWorks ecosystem.

    3. The first two items in your list of Python's advantages (subroutines and sharing code across experiments) are addressed by MWEL. Specifically, it supports function-like macros and "include" statements. Again, I will try to provide examples very soon.

    Cheers,
    Chris

  5. 5 Posted by mhisted on 24 Jun, 2017 04:13 PM

    mhisted's Avatar

    Hi Chris,

    Yes, I think this will be a conversation that will take a bit of time
    before we come to any conclusion one way or the other. I appreciate your
    expertise in implementing MWorks and I'm sensitive to your concerns about
    maintainability and how much effort any Python changes might be. But since
    we're talking about a proposal that's reasonably long-term, I thought it
    would make sense to bring others who use MWorks into the conversation. I
    suspect that I have relatively specific needs (perhaps also shared by
    Lindsey) since we deploy MWorks code across more than a dozen training
    rigs, and thus we perhaps need more automation than most. So I think it's
    worth me hearing more about other users' needs and what you see as the best
    path forward. I do think that the Python proposal is largely independent
    of the DSL/XML - sounds like MWEL is close to done, and we would need to
    depend on backwards-compat of a lot of old XML code.

    More next week.

    thanks,
    Mark

  6. Support Staff 6 Posted by Christopher Sta... on 27 Jun, 2017 05:35 PM

    Christopher Stawarz's Avatar

    Hi Mark,

    Here are the promised examples.

    To produce these, I started with an existing, XML-based MWorks experiment (analog_io.xml). This is actually a test of MWorks' NIDAQ interface, which gets run as part of the nightly test suite. The details of what it does aren't important, but note that it includes

    1. an I/O device with channels;
    2. a stimulus that updates dynamically, based on input from the I/O device; and
    3. a task system with multiple states.

    Hence, it's short but somewhat sophisticated.

    I then rewrote the example in both Python (analog_io.py) and MWEL (analog_io.mwel). The Python version is my best guess at what a Python-based MWorks experiment would look like. You may have other ideas about how it should look; if so, I'd like to hear about them. The MWEL version is working code; I can load and run it just like the XML version, and it performs identically.

    In the Python code, I've included comments pointing out various issues I see. The MWEL version has none of these issues.

    Please have a look and let me know what you think.

    Chris

  7. 7 Posted by mhisted on 28 Jun, 2017 02:04 PM

    mhisted's Avatar

    Hi Chris,

    I think the MWEL seems like a big step forward over XML. I don't have any major comments, though I guess I'd prefer 'transition' over 'goto' in some cases.

    To move forward the discussion of having a parallel Python approach to defining experiments (supported along with the XML/MWEL), I wanted to also provide an example of what a 'MWEPy' experiment might look like.
    I put together something that can serve as a start for discussion (attached).

    I'd think the next step (no rush) would be for you to make comments on the python syntax and then perhaps we can discuss.

    -M

  8. Support Staff 8 Posted by Christopher Sta... on 21 Jul, 2017 03:43 PM

    Christopher Stawarz's Avatar

    Hi Mark,

    A few updates:

    • MWEL support is now in the nightly build
    • I've published an in-progress snapshot of the MWorks 0.8 docs, which include an MWEL reference

    I'll share some comments on your MWorks/Python example soon.

    Cheers,
    Chris

  9. Support Staff 9 Posted by Christopher Sta... on 31 Jul, 2017 03:39 PM

    Christopher Stawarz's Avatar

    Hi Mark,

    I've attached a copy of your "MWPL" example with some comments added (all prefaced with "CJS"). Some additional questions:

    • How would you implement an experiment like this, where the position of the stimulus is determined by expressions that are evaluated every time it's drawn? Would you just write those expressions as strings in Python?

    • How do you imagine stopping or pausing an experiment inside a state? Stopping might be implemented by the MWorks' action wrappers throwing an exception if the experiment is stopped. However, I don't know how we'd pause and restart a Python function in the middle, short of delving deep into the guts of the Python interpreter and adding support for it, somehow.

    • Is this something you might someday want to use on iOS? If so, we'd need to bundle Python with MWorks on that platform and deal with whatever porting issues come up. (MWEL neatly avoids this by converting to XML on the client side.)

    I think most of the work in implementing support for Python-based experiments is in exposing the basic elements of experiment construction to Python. Specifically, I mean providing the means to create, connect, and control components via Python. The biggest issue is that most of the details of experiment construction are handled in the XML parser, and breaking them out of there would need to be handled with great care (and may require some substantial rethinking/reworking of MWorks' internals). Another issue, which I noted in my Python mockup, is that the "guts" of actions aren't really accessible as simple functions, so they'd need to be reworked.

    Once that work is done, then it should be relatively straightforward to implement MWPL on top of it. And it could be useful for other things, too. For example, the MWEL processor could construct the experiment directly, instead of converting to XML and handing it off to the XML parser.

    I don't know how good a time estimate I can put on this work -- maybe a couple months? However, mucking with the XML parser is definitely dangerous territory, with high potential for things getting busted in subtle ways. Again, great care (and much testing) would be needed.

    Chris

  10. 10 Posted by mhisted on 01 Aug, 2017 01:58 AM

    mhisted's Avatar

    Hi Chris,

    I revised the Python a bit based on your comments. New proposal is attached.
    Notable:
    - I proposed a new way to keep track of MW variables and stimuli inside their own namespace. Initialization is now just creating a Python object and assigning to a field a namespace/mapping object. Is something like this possible?
    - Shared python data across state system States, like local vars (not MW vars) can live in the top-level namespace.
    - I'd propose using the python Exception/Warning/Logging mechanism and displaying those messages in the Server console.
    (Perhaps: embed a Python interpreter. import user script inside a try: except, and call a particular method to run the experiment.)

    Additional comment:
    I now minimally use the "live update" mechanism, in part because it's been unclear to us when various MW elements read
    the changed values. I could have gone through with you or with the code, but we haven't done this.
    As I understand, the usualy way JS/Python deal with 'live updates' is invoking callbacks at particular well-defined times.
    So an alternative to the live update mechanism would be to add a callback to each stimulus:

    def cbFrameRect(stim, curr_ms, time_to_next_frame_ms): # or whatever other vars are useful; called each frame
            stim.set_position(x_position=curr_ms*10, y_position=cursor_y)
    mw.stim.rect = mw.stimulus.Rectangle(x_size=cursor_size, y_size=cursor_size,
                                   x_position=cursor_x, y_position=cursor_y, frame_callback=cbFrameRect)

    Responses to your questions:

    Stopping/pausing: We don't use pause now, but I understand the desire to implement it if others use it. I guess I don't fully understand what people use it for. Stopping by throwing an exception sounds good. The question here is how to preserve state across stop/start as the naive thing is to reload the script on each start.

    iOS: we haven't even looked at iOS, and as we discussed Apple seems to be signalling they care at least a little about MacOS and Mac hardware. Again I can appreciate other users may want this.

    > Once that work is done, then it should be relatively straightforward to implement MWPL on top of it. And it could be useful for other things, too. For example, the MWEL processor could construct the experiment directly, instead of converting to XML and handing it off to the XML parser.
    >
    > I don't know how good a time estimate I can put on this work -- maybe a couple months? However, mucking with the XML parser is definitely dangerous territory, with high potential for things getting busted in subtle ways. Again, great care (and much testing) would be needed.
    >

    It seems like these changes could have other benefits, but I take your point that this is a major decision. Let's see if we can agree about the syntax details before committing to anything.

    My guess is the live-update / callback question is a bit too big for email only and best via phone - perhaps sometime next week?

    thanks,
    Mark

  11. 11 Posted by mhisted on 07 Aug, 2017 12:47 AM

    mhisted's Avatar

    Hi Chris,

    > • How would you implement an experiment like this, where the position of the stimulus is determined by expressions that are evaluated every time it's drawn? Would you just write those expressions as strings in Python?

    I've thought a bit more about how to update stimulus attributes during an experiment. My last was a little discombobulated.

    We're talking about this type of definition from your python example, I believe:

    > target = mw.stimulus.fixation_point('target',
    > y_size = target_size,
    > x_position = '-amplitude')
    > # amplitude is a mworks variable

    I see a few ways to handle this. I agree eval'ing strings is not a great approach.
    First, passing a callable as a parameter value. That callable would be called anytime the value is needed. (Pros and cons at bottom of email):

    > target = mw.stimulus.fixation_point('target',
    > y_size = 10,
    > x_position = lambda: -mw.vars.amplitude)
    > # 3 lines
    > # mw.vars is the variable dictionary exposed by the mworks module

    Second, using a callback function passed to the stimulus object:
    > def fn(self, frame_ms):
    > self.x_position = -mw.vars.amplitude
    > target = mw.stimulus.fixation_point('target',
    > y_size = 10,
    > x_position = None,
    > frame_callback = fn)
    > # 6 lines

    Third, a class that inherits from the stimulus class to override a callback method:
    > class xMovingFp (mw.Stimulus.FixationPoint):
    > def frame_fn(self, frame_ms):
    > self.x_position = -mw.vars.amplitude
    > mw.stim.target = xMovingFp( # uses my preferred stim def syntax, assigning instance to mw.stim attribute
    > y_size = 10,
    > x_position = None)
    > # 6 lines
    > # scoping of mw.vars should be fine, I think.
    > # Pro: easy to document the various callbacks that can change instance attributes

    Pros for 2, 3
    - (vs 1 and also passing a mw.Variable and letting the stimulus read it when needed): makes the update time where param values are read explicit.
    - (vs 1): Fewer function calls, as the callable parameter needs to be called anytime the value is used; the callback is called at defined times, like once a frame.
    Cons for 2, 3:
    - code is more verbose
    Cons for all:
    - perhaps requires fundamental changes in mworks?

    Of the three I think I prefer #3, as I think it's least surprising for python users, and would be easy to document the different functions: "All stimulus types have a frame_fn and a trial_start_fn method that can be used to..."

    Also, I believe with this approach (any of 1,2,3) we can drop Variable.value and just say that a mw.Variable always evaluates to a numeric value (float/int?) when evaluated in a numeric context. And when MW classes (stimuli, variables, etc.) are instantiated, they evaluate any Variables. Then, the only way to change stimulus attributes is explicit --- a callable.

    I think this is a slight mismatch with the current MWorks architecture. But I'm not sure it's a big mismatch. With MWorks today, the stimulus doesn't actually use the new variable value till redraw, so the real-world action is temporally decoupled from the variable change that generates an event. If we log (or throw off an event) on redraw, I think we capture the same conceptual functionality as today.

    Comments?

    Mark

  12. Christopher Stawarz closed this discussion on 24 Jul, 2018 01:21 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac