Simulation with Resultfile or Flags Fail to execute

Description

When calling
OMPython.simulaute(resultfile='/home/user/modelica/testfile0.mat')
with a result file (and most likely also with other flags, the following log message is shown

Failed to simulate simulation: [Errno 2] No such file or directory: '/tmp/tmp37u5_obx/Package.Simulation -r=/home/user/modelica/testfile0.mat'

Steps to reproduce

Minimal Example that fails

from os.path import expanduser
from OMPython import ModelicaSystem

home_dir = expanduser("~")

model_path = home_dir + '/modelica/resultfilebug/'
model_file_name = 'BouncingBall.mo'
model_name = 'BouncingBall' 
mod = ModelicaSystem(model_path + model_file_name, model_name )

mod.buildModel()

result_file_path = home_dir + '/modelica/resultfilebug/resultfiles/'
result_file_name = 'resultfile_0.mat'

mod.simulate(resultfile=result_file_path+result_file_name)

Throws followingexception:

Exception has occurred: FileNotFoundError (note: full exception trace is shown but execution is paused at: _run_module_as_main)
[Errno 2] No such file or directory: '/tmp/tmpj5k_1p6k/BouncingBall -r=~/modelica/resultfilebug/resultfiles/resultfile_0.mat'

Even though the file exists in my file system

ls /tmp/tmpqmbz5d57/BouncingBall
/tmp/tmpqmbz5d57/BouncingBall

Running

user@machine:~$ /tmp/tmpqmbz5d57/BouncingBall -r=/home/user/modelica/resultfilebug/resultfiles/resultfile_0.mat

LOG_ASSERT | debug | simulation_input_xml.c: Error: can not read file BouncingBall_init.xml as setup file to the generated simulation code.
Execution failed!

While

user@machine:/tmp/tmpq5rla9at$ ./BouncingBall -r=/home/user/modelica/resultfilebug/resultfiles/resultfile_0.mat

LOG_SUCCESS | info | The initialization finished successfully without homotopy method.
LOG_SUCCESS | info | The simulation finished successfully.

Expected behavior

The model should just build and return the following log messages

LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.

Screenshots

If applicable, add screenshots to help explain your problem.

Version and OS

Python Version: Python 3.10.12
OMPython Version: 3.5.1
OpenModelica Version: 1.22.3
OS: Windows 11 64 bit - Code run from WSL :Ubuntu 22.04.4 LTS

Additional context

When digging into the problem and debugging I came across this:

The error message suggests that the subprocess is unable to find the executable file /tmp/tmpqmbz5d57/BouncingBall -r=/home/user/modelica/testfile0.mat

The issue here is that the provided cmd is being interpreted as a single command with spaces in its name. Instead, you should separate the executable path and the command-line arguments into separate elements of the cmd list.

So the problem is created within

p = subprocess.Popen(cmd, env=my_env)

A solution to the problem is how cmd is created. If it is a list with

getExeFile
override
csvinput
r
simflags

as seperate list entries, it works with my system configuration

I also found a working solution that made the code work for my system.

Before I make a pull request I'm wondering the following.
What format are the simFlags expected to be ?
A long string containing the flags like
mod.simulate(simflags='-flag1 -flag2 -...')
or a list containing the flags as sting
mod.simulate(simflags=['-flag1', '-flag2' '-...'])

I couldn't really find an answer to that in the documentation.
I can make a solution that works for both cases, but that might be unnecessary complicated if just on of them is a valid input.

fixes issue OpenModelica#206

Hi, I have the same problem, I have found out that OMPython creates a command like this

cmd = '/tmp/tmpltwurgv3/variable_torque -overrideFile=/tmp/tmpltwurgv3/variable_torque_override.txt -r=variable_torque_size_50.mat'  
subprocess.Popen(cmd)

But doing the following test works nice

import subprocess
cmd = [
    "/tmp/tmpltwurgv3/variable_torque",
    "-overrideFile=/tmp/tmpltwurgv3/variable_torque_override.txt",
    "-r=variable_torque_size_50.mat"
]
p = subprocess.Popen(cmd)

So I suppose that the format of simFlags are like this mod.simulate(simflags=['-flag1', '-flag2' '-...'])

I have the same problem.

@InigoGastesi

@InigoGastesi
Yes this was also the solution i found and how i fixed i in the commit i made.
If you give the subprocess a python list containing the path to the executable at index 0 and then all the flags in the subsequent indices, it works. Compared to the current implementation you only have to remove the initial blank space from the flags.

@NilsDressler

@NilsDressler I using your commit now in my project, thank you

@NilsDressler

@NilsDressler the issue is fixed with this commit fa56e30 and thanks for @NilsDressler for fixing the issue

@arun3688

Hi @arun3688, I think that this issue is not fixed in that commit. In that commit the only think that is fixed is the problem with the subprocess

@InigoGastesi

@InigoGastesi Yes that is true, but still the issue fixes the subprocess problem in linux.
@NilsDressler It is not allowed to provide result file path in the simulate() API, you can only provide the name for the result file and the result file will always be generated in the temp directory. see the docs here https://openmodelica.org/doc/OpenModelicaUsersGuide/1.22/ompython.html#simulation

The model can be simulated using the simulate API in the following ways,

without any arguments
resultfile (keyword argument) - (only filename is allowed and not the location)
simflags (keyword argument) - runtime simulationflags supported by OpenModelica

An example of how to use the simflags is also provided in the documentation

@arun3688

@arun3688 I'm checking your last comment and in the following code

filename = f'{model_name}_size_{size}'
print(filename) # voltage_torqueVariable_size_50
mod.simulate(resultfile=f'{filename}.mat') #Para simular el modelo, empleo el método ".simulate()"

I have this error

[Errno 2] No such file or directory: '/tmp/tmp9y_av4ml/voltage_torqueVariable -overrideFile=/tmp/tmp9y_av4ml/voltage_torqueVariable_override.txt -r=voltage_torqueVariable_size_50.mat'

I checked it and the file exists. Also, I tried do the same without .mat extension and the same problem appears. As you can see in the print there is only filename

@InigoGastesi

@InigoGastesi what happens if you directly use mod.simulate(resultfile="voltage_torqueVariable_size_50.mat")

Same error :(

@InigoGastesi

@InigoGastesi strange what happens if you just run mod.simulate()

@InigoGastesi

@InigoGastesi @NilsDressler I can reproduce the issue, I will fix it today.

@NilsDressler

@NilsDressler @InigoGastesi the issue is fixed with this commit 69a502e
Actually it is possible to provide result file path in the as said in the documentation simulate(resultfile="C:/resultFile/test.mat") API, see the example below, update your package using python -m pip install -U https://github.com/OpenModelica/OMPython/archive/master.zip

An example usage is

Usage

>>> from OMPython import ModelicaSystem
>>> mod = ModelicaSystem("C:/BouncingBall.mo", "BouncingBall")
>>> mod.simulate(resultfile="C:/OPENMODELICAGIT/BouncingBall_res.mat") // path provided by user
LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.
>>> mod.resultfile
'C:/OPENMODELICAGIT/BouncingBall_res.mat'
>>> mod.getSolutions()
('der(h)', 'der(v)', 'e', 'flying', 'foo', 'g', 'h', 'impact', 'time', 'v', 'v_new')
>>> mod.simulate(resultfile="test.mat") // providing only the file name 
LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.
>>> mod.getSolutions()
('der(h)', 'der(v)', 'e', 'flying', 'foo', 'g', 'h', 'impact', 'time', 'v', 'v_new')

@NilsDressler

@NilsDressler please check with the nightly build and close the ticket if it works for you. Thanks!

I'm still experiencing issues with this:

self = <Popen: returncode: 255 args: '/tmp/tmpu48lhhg0/myTestPackage...>
args = ['/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1 -overrideFile=/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1_override.txt']
executable = b'/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1 -overrideFile=/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1_override.txt'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None
startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1, errwrite = -1
restore_signals = True, gid = None, gids = None, uid = None, umask = -1
start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        gid, gids, uid, umask,
                        preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1 -overrideFile=/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1_override.txt'

@j-emils

@j-emils I can see that you are directly using the subprocess to execute the executable and the error message says it cannot find the executable in that location, also i see you are not using ModelicaSystem constructor, It would be good if you can provide a minimal example to reproduce the issue, I suggest you to change to the executable directory before using the subprocess command

@arun3688

@arun3688, we are using the ModelicaSystem constructor, it's just that we have created a wrapper around it. Maybe there's a dependency on the OpenModelica version? It's failing with version 1.22.0.

@j-emils

@j-emils please uninstall your OMPython package and install it with the latest master,

pip uninstall OMPython
and then
pip install git+https://github.com/OpenModelica/OMPython.git/@master

@arun3688

@arun3688, I'm still getting the same error. If I change the call with subprocess to use a list instead of a string it works, analogous to the proposal from @NilsDressler . Will try to create an example for you to test.

@casella

@casella If the latest nightly referred to 891528c then it still does not work for my minimal example.
I will make a pull request for the solution provided by me

@arun3688

@arun3688 can you please follow this up? Thanks!

@casella

@casella the issue is fixed with the this commit #214

@arun3688

Great, thanks @arun3688!

Navigation Menu

Repository navigation

Simulation with Resultfile or Flags Fail to execute #206

Description

Steps to reproduce

Expected behavior

Screenshots

Version and OS

Additional context

5 remaining items

Usage

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Simulation with Resultfile or Flags Fail to execute #206

Description

Description

Steps to reproduce

Expected behavior

Screenshots

Version and OS

Additional context

Activity

NilsDressler commented on Apr 19, 2024

InigoGastesi commented on Apr 23, 2024

j-emils commented on Apr 23, 2024

NilsDressler commented on Apr 23, 2024

InigoGastesi commented on Apr 24, 2024

arun3688 commented on Apr 24, 2024

InigoGastesi commented on Apr 24, 2024

arun3688 commented on Apr 24, 2024

InigoGastesi commented on Apr 24, 2024

arun3688 commented on Apr 24, 2024

InigoGastesi commented on Apr 24, 2024

arun3688 commented on Apr 24, 2024

5 remaining items

arun3688 commented on Apr 26, 2024

arun3688 commented on Apr 26, 2024

Usage

casella commented on May 2, 2024

j-emils commented on May 6, 2024

arun3688 commented on May 6, 2024

j-emils commented on May 6, 2024

arun3688 commented on May 6, 2024

j-emils commented on May 6, 2024

NilsDressler commented on May 6, 2024

casella commented on May 7, 2024

arun3688 commented on May 7, 2024

casella commented on May 7, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions