Standardize how to do environment variable parsing & substitution #18307

There're two parts to this issue:

Parsing

We should probably always follow dotenv syntax:

• Circuit Python: https://docs.circuitpython.org/en/latest/shared-bindings/dotenv/index.html#
  It's almost very similar to what we do; except for the multiline support and comments on the same line, so good news is it won't break us if we decide to align.

• Python dot-env: https://pypi.org/project/python-dotenv/
  Match with how it does variable substitution: Standardize how to do environment variable parsing & substitution #18307

Substitution

There are currently two ways of substituting variables in a .env file:

VAR2="value;${env:VAR1}"

and

VAR2="value;${VAR1}"

and a combination of both has to used to resolve all variables in different situations. However, this is not standard syntax. Again we should probably rely on the dotenv syntax always.

• ${env:...} only works for process env variables (for eg. PATH) when debugging or when resolving env variables in settings.
  • Exception: There's a bug due to which debugger doesn't resolve ${env:...} in .env files when debugging a Python file: Environment variable definitions file substitution does not work when debugging a Python file vscode-python-debugger#159. Make sure Environment variable definitions file substitution does not work when debugging a Python file vscode-python-debugger#159 is taken care of with this issue.
• "Recursive" variable substitution does not work with ${env:...}. It works only with ${...}. Need to correct the docs: Test envVarsService newline substitution behavior #17747 (review).
• .env file is currently not supported when running code in terminal: Use environment variables defined in .env file when running code in a terminal #944
• ${...} can not be used for process env variables when debugging: https://github.com/microsoft/debugpy/issues/821. Solution:
  • Debugging doesn't use environment variable substitution/resolution support in the extension. Expose

    vscode-python/src/client/common/variables/environmentVariablesProvider.ts

    Line 56 in 456ac06

    public async getEnvironmentVariables(resource?: Uri): Promise<EnvironmentVariables> {

    as an API and have the debugger use it to fetch env variables.
  • Follow the standard python-dotenv syntax to resolve process env variables in debugger: https://pypi.org/project/python-dotenv/, i.e add support to resolve using ${...} in addition to ${env:...}.