Skip to content

Question: can we filter metadata? #65

Closed

Description

@FixRM

Hello!

Thank you for a great tool! We use it to generate docs for Microsoft Common Data Service OData endpoint (formerly Dynamics CRM, formerly D365). This endpoint is generic and has quite big CSDL file (about 4 MB). Resulting JSON file is about 880 MB!!! Can you please suggest how we can cut some unused datasets and related stuff before generating OpenAPI file?

Activity

  1. xuzhg commented on Jun 1, 2020

    @xuzhg
    Contributor

    Artem Grunin (@FixRM) Thanks for trying this tool.

    1. there are some settings that you can use to do filter. For example: if set EnableNavigationPropertyPath = false, there will be no navigation property path.

    2. IODataPathProvider has the "CanFilter" that is designed for your scenario, but so far, i don't have much requirement, so i haven't exposed it throw "ODataContext".

  2. FixRM commented on Jun 1, 2020

    @FixRM
    Author

    Sam Xu (@xuzhg), thanks for quick reply. Can you please give me a sample of how to use IODataPathProvider?

  3. xuzhg commented on Jun 1, 2020

    @xuzhg
    Contributor

    Artem Grunin (@FixRM) My design is that you can derive from "ODataPathProvider" and override this virtual method. However, "ODataPathProvider" is used in the ODataContext, that's an internal class.

    I'd like to expose an interface in the setting class so developer can create an instance of "his own ODataPathProvider", but it's not supported yet.

  4. FixRM commented on Jun 1, 2020

    @FixRM
    Author

    I see. What other options do I have? I was thinking of filtering while loading XML nodes, but it seems tricky and my VS is dying when I try to load original file into it

  5. xuzhg commented on Jun 1, 2020

    @xuzhg
    Contributor
    1. use the Capabilities annotation, but this way will change your CSDL
    2. I expose the Provider at the setting for you to set. Need some time to implement it.
  6. darrelmiller commented on Jun 3, 2020

    @darrelmiller
    Member

    If we removed examples as proposed here #68 it would help to reduce the document size.

    Also, yaml produces a much more compact format than JSON.

  7. darrelmiller commented on Jul 4, 2020

    @darrelmiller
    Member

    Artem Grunin (@FixRM) I think your best option would be to run the CSDL through an XSLT transform first to cut out the parts of the CSDL you don't want to process. We do some preprocessing of our CSDL this way using this approach https://github.com/microsoftgraph/msgraph-metadata/blob/master/transforms/csdl/preprocess_csdl.xsl

  8. FixRM commented on Jul 9, 2020

    @FixRM
    Author

    Darrel (@darrelmiller), thank you for your suggestion. Can you please help me with the transform?

    My CSDL looks like this:

    <edmx:Edmx Version="4.0">
        <edmx:Reference Uri="http://vocabularies.odata.org/OData.Community.Keys.V1.xml">
        <edmx:Include Namespace="OData.Community.Keys.V1" Alias="Keys"/>
        <edmx:IncludeAnnotations TermNamespace="OData.Community.Keys.V1"/>
    </edmx:Reference>
    <edmx:Reference Uri="http://vocabularies.odata.org/OData.Community.Display.V1.xml">
        <edmx:Include Namespace="OData.Community.Display.V1" Alias="Display"/>
        <edmx:IncludeAnnotations TermNamespace="OData.Community.Display.V1"/>
    </edmx:Reference>
    <edmx:DataServices>
        <Schema Namespace="Microsoft.Dynamics.CRM" Alias="mscrm">
            <EntityType Name="crmbaseentity" Abstract="true"/>
            <EntityType Name="accountleads" BaseType="mscrm.crmbaseentity">
                        ***
    

    I started with this one, but it generates same document.

    <?xml version="1.0" encoding="utf-8"?>
    <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
      
      <xsl:output method="xml" indent="yes"/>
      <xsl:template match="@* | node()">
        <xsl:copy>
          <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
      </xsl:template>
    
      <xsl:template match="Schema/EntityType[not(@Name='account')]" />
      
    </xsl:stylesheet>
    

    What am I doing wrong?

  9. FixRM commented on Jul 9, 2020

    @FixRM
    Author

    It was namespace problem. Correct transform:

    <?xml version="1.0" encoding="utf-8"?>
    <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                    xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx"
                    xmlns:edm="http://docs.oasis-open.org/odata/ns/edm"
                    xmlns="http://docs.oasis-open.org/odata/ns/edm">
      <xsl:strip-space elements="*"/><!-- Remove empty space after deletions. -->
      <xsl:output method="xml" indent="yes"/>
      <xsl:template match="@* | node()">
        <xsl:copy>
          <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
      </xsl:template>
    
      <!--Remove unused entities-->
      <!--<xsl:template match="edm:Schema/edm:EntityType[not(@Name='account')]" />-->
    
      <!--Remove unused entity sets-->
      <xsl:template match="edm:Schema/edm:EntityContainer/edm:EntitySet[not(@Name='accounts')]" />
    
      <!--Remove unused actions-->
      <!--<xsl:template match="edm:Schema/edm:Action" />-->
    
      <!--Remove unused types-->
      <!--<xsl:template match="edm:Schema/edm:ComplexType" />-->
    
      <!--Remove unused functions-->
      <!--<xsl:template match="edm:Schema/edm:Function" />-->
    
      <!--Remove unused enums-->
      <!--<xsl:template match="edm:Schema/edm:EnumType" />-->  
    
    </xsl:stylesheet>
    
  10. bkwdesign commented on Dec 16, 2021

    @bkwdesign

    Artem Grunin (@FixRM) My design is that you can derive from "ODataPathProvider" and override this virtual method. However, "ODataPathProvider" is used in the ODataContext, that's an internal class.

    I'd like to expose an interface in the setting class so developer can create an instance of "his own ODataPathProvider", but it's not supported yet.

    I also have an EDMX that produces over 6400 paths- and 1800+ schemas -my swagger result is only 17 MB, but I’m also interested in ways to pass in a schema-of-interest and have the rest ignored

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions